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Executive  Summary 


The  Interactive  Model-Centric  Systems  Engineering  (IMCSE)  research  program  arises  from  the 
unique  opportunity  to  investigate  the  various  aspects  of  humans  interacting  with  models  and 
model-generated  data,  in  the  context  of  systems  engineering  practice.  IMCSE  research  aims  to 
develop  transformative  results  through  enabling  intense  human-model  interaction,  to  rapidly 
conceive  of  systems  and  interact  with  models  in  order  to  make  rapid  trades  to  decide  on  what 
is  most  effective  given  present  knowledge  and  future  uncertainties,  as  well  as  what  is  practical 
given  available  resources  and  constraints.  While  model-based  engineering  initiatives  are 
advancing  technical  aspects  of  models  in  the  engineering  of  systems,  this  research  advances 
knowledge  relevant  to  human  interaction  with  models  and  model-generated  information. 

Figure  1  highlights  several  questions  the  research  seeks  to  address. 


Significant  progress  on 
theory/practice 
of  model-based  systems 
engineering 


...  insufficient  focus  on 

human-model 

interaction 


How  do  humans  interact  with  models 
and  model-generated  information? 

How  do  humans  interact  with  each  other 
using  models? 

What  cognitive  challenges  exist  for 
model-informed  decision-making  ? 

What  are  essential  human  roles  in 
model-centric  environments? 

How  can  interactivitity  of  humans  and 
models  be  made  more  effective? 


Figure  1  Areas  of  Inquiry 

This  report  discusses  outcomes  of  Phase  5  of  the  IMCSE  research  program,  focusing  on  three 
areas: 

Human-Model  Interaction.  The  empirical  study  on  model-centric  decision  making,  initiated  in 
Phase  4,  was  completed,  and  feedback  was  obtained  in  several  technical  exchanges  with  the 
practitioner  community.  The  empirical  study  findings,  analogy  studies,  and  other  secondary 
source  data  were  used  to  develop  a  set  of  guiding  heuristics.  Several  validation  activities  were 
conducted,  towards  the  final  goal  of  developing  a  validated  set  of  guiding  principles  for 
effective  human-model  interaction.  State-of-the-practice  findings  were  used  as  a  basis  for 
conducting  a  technical  exchange  workshop  with  research  stakeholders.  The  workshop  provided 
research  feedback  and  identified  needs  for  improving  the  state-of-the-practice  for  human- 
model  interactivity. 
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Interactive  Epoch-Era  Analysis.  Research  matured  the  approach  for  evaluating  systems  under 
dynamic  uncertainty.  The  extended  framework  for  interactive  capability  and  scaling  to  big  data 
was  completed,  including  several  interactive  prototypes  for  demonstration  purposes. 
Demonstration  cases  were  completed  for  a  space  tug  multi-mission  orbital  transfer  vehicle  and 
a  commercial  offshore  ship.  The  framework  can  be  applied  for  both  modular  and  non-modular 
solutions  for  comparison  within  a  tradespace  (which  helps  to  assess  the  lay-in,  opportunity,  and 
carrying  costs  of  modularity),  and  as  a  means  for  determining  the  costs  and  benefits  of  having 
the  option  to  switch  between  states  (e.g.  swapping  a  mission  payload  or  manufacturing  line). 
IEEA  involves  both  visualization  and  interaction  to  varying  degrees.  An  experiment  to 
investigate  the  impacts  of  visualization  and  interaction  in  a  decoupled  manner,  initiated  in 
Phase  4,  was  completed,  with  several  findings.  A  set  of  knowledge  assets  was  completed 
including  the  Interactive  EEA  framework,  demonstration  prototypes,  case-based  impact  studies, 
and  results  of  the  experiment. 

Curation  of  Model-Centric  Environments.  Prior  phase  research  reinforced  the  interest  and 
need  for  organizations  to  perform  a  model  curation  activities.  Under  the  premise  that  model¬ 
centric  environments  of  the  future  will  necessitate  specialized  leadership  and  competencies,  a 
new  leadership  role  for  curation  was  further  investigated.  Four  stages  were  identified  for 
organizations  transforming  under  the  digital  paradigm.  Specific  needs  were  identified  for 
establishing  leadership  and  practices  for  model  curation  through  primary  and  secondary 
sources.  Investigations  suggest  seven  different  types  of  organizations  for  implementing  model 
curation,  based  on  different  needs  and  organization  forms.  Research  findings  were  shared  with 
research  stakeholders  as  a  preliminary  validation  of  needs  and  emerging  practices.  The  need  for 
an  instrument  for  organizations  to  assess  their  model  leadership  capabilities  and  risks  was 
confirmed  in  a  technical  exchange  workshop  and  other  meetings  with  research  stakeholders. 

IMCSE  research  was  presented  and  discussed  with  practitioners  and  sponsors  in  numerous 
research  meetings  and  workshops  throughout  Phase  5.  These  included  SERC  events,  NDIA  SE 
Conference,  INCOSE  International  Workshop,  and  various  meetings  at  MIT  and  other  research 
stakeholder  locations.  These  interactions  have  provided  valuable  feedback  and  inputs  to  the 
research  activities,  and  served  to  discover  relevant  research  ongoing  in  the  larger  systems 
community. 
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Interactive  Model-Centric  Systems  Engineering  (IMCSE)  Phase  5  Research 


Models  are  increasingly  used  to  drive  major  acquisition  and  design  decisions  (Zimmerman  et  al., 
2017),  yet  model  developers,  analysts,  and  decision  makers  are  faced  with  many  challenges. 
The  systems  community  has  made  progress  on  standards,  methods  and  techniques  for  model- 
based  engineering,  yet  little  focus  has  been  given  to  complexities  of  human-model  interaction. 

Interactive  Model-Centric  Systems  Engineering  (IMCSE)  research  generates  knowledge 
impacting  human  effectiveness  in  model-centric  environments  of  the  future  (Rhodes  &  Ross, 
2016).  The  Phase  5  research  provided  the  opportunity  to  build  on  the  research  outcomes  of  the 
prior  phases.  This  report  discusses  the  findings  of  three  areas  of  focus: 

1.  Human-Model  Interaction 

2.  Interactive  Epoch-Era  Analysis 

3.  Curation  of  Model-Centric  Environments 
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Human-Model  Interaction 


Addressing  complex  systems  problems  requires  both  human  intelligence  and  the  use  of  digital 
system  models.  Open  areas  of  inquiry  include:  how  individuals  interact  with  models;  how 
multiple  stakeholders  interact  using  models  and  model  generated  information;  facets  of  human 
interaction  with  visualizations  and  large  data  sets;  and  underlying  fundamentals  such  as  model 
purpose  and  model  handling. 


Background 

The  research  is  based  on  the  belief  that  human-model  interaction  needs  to  be  specifically 
considered,  given  models  are  an  abstraction  of  reality  and  there  are  unique  factors  and 
considerations.  A  science  of  human-systems  integration  (HSI)  has  emerged,  yet  focus  is  on 
humans  within  operational  systems,  while  models  are  abstractions  of  reality.  The  relatively 
mature  field  of  human-computer  interaction  (HCI)  offers  valuable  insights,  however  focus  is  on 
designing  computer  interfaces.  A  report  of  a  recent  workshop  sponsored  by  the  National 
Science  Foundation  (NSF),  the  National  Aeronautics  and  Space  Administration  (NASA),  the  Air 
Force  Office  of  Scientific  Research  (AFOSR),  and  the  National  Modeling  and  Simulation  Coalition 
(NMSC),  highlights  the  need  for  understanding  the  individuals  involved  in  the  modeling  process 
and  how  these  individuals  affect  model  development  and  usage  (NSF,  2016). 

Recent  research  studies  have  provided  insight  into  the  "state  of  the  practice"  on  human-model 
interaction  and  decision  making  with  models  (Rhodes  &  Ross,  2016,  German,  2016).  As  shown 
in  Figure  2,  there  are  three  elements  involved  in  model-centric  decision  making:  the  decision  to 
be  made,  the  digital  thread/digital  system  model,  and  the  human  actors.  The  focus  of  this  study 
is  on  the  human  actors. 


Figure  2  Elements  of  Model-Centric  Decision  Making 
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One  activity  in  this  research  program  is  investigation  of  guiding  principles  for  human-model 
interaction.  The  principles  are  expected  to  result  from  a  formal  iterative  evaluation,  refinement 
and  testing  of  the  heuristics  that  have  been  identified  through  a  variety  of  research  activities. 


Research  Approach 

Our  research  on  human-model  interaction  has  investigated  the  literature  across  various  fields 
including  cognitive  science,  decision  analysis,  organizational  and  human  behavior,  and  systems 
science.  Several  secondary  source  research  studies  provided  "analogy  case"  insights  from 
situations  having  similarities  to  the  transition  from  traditional  engineering  environments  to 
model-centric  engineering  environments  (German  &  Rhodes,  2016;  German,  2017).  Knowledge 
was  also  drawn  from  related  fields  of  human-systems  integration  (HSI),  human  computer 
interaction  (HCI),  and  Visual  Analytics  (VA),  as  relevant  to  the  research  (Pew  &  Mavor,  2007; 
Shneiderman  et  al.,  2016;  Thomas  &  Cook,  2005). 

Key  research  themes  emerged  from  these  investigations,  in  addition  to  themes  identified 
through  a  pathfinder  research  workshop  involving  expert  practitioners  and  researchers  (Rhodes 
&  Ross,  2015).  An  exploratory  interview-based  study  of  model-centric  decision  making  resulted 
in  findings  drawn  from  thirty  subject  matter  experts  (German  &  Rhodes,  2017).  Candidate 
heuristics  were  derived  primarily  from  the  expert  interviews,  as  supported  by  insights  from  the 
literature  investigation  and  second  source  studies.  A  series  of  individual  and  group  activities 
were  used  for  preliminary  validation  and  refinement  of  heuristics. 


Importance  of  Endogenous  Perspective 

Investigation  of  human-model  interaction  and  model-centric  decision  making  has  underscored 
the  need  to  view  humans  as  endogenous  constituents  within  model-centric  engineering 
environments.  The  endogenous  point  of  view,  as  asserted  by  Forrester,  starts  with  the  question 
of  where  is  the  boundary  "that  encompasses  the  smallest  number  of  components,  within  which 
the  dynamic  behavior  under  study  is  generated?"  (Richardson,  2011). 

By  drawing  the  boundary,  models  and  human  actors  are  designated  as  constituents  within  the 
environment.  This  contrasts  with  how  humans  are  often  viewed  as  being  "on  the  loop", 
interacting  with  the  environment  as  an  external  agent.  Models  represent  an  abstraction  of 
reality,  and  can  come  in  a  variety  of  forms  and  formats,  but  fundamentally  they  are  an 
encapsulation  of  reality  that  humans  use  to  augment  their  ability  to  make  sense  of  the  world 
and  anticipate  future  outcomes.  The  idea  that  "humans  use"  models,  highlights  human 
interaction  as  a  necessary  factor  within  all  models. 

In  the  emerging  model-centric  practice,  model  and  humans  are  intertwined.  Models  are  used 
by  humans  and  that  interaction  influences  how  models  are  conceived  and  used  in  decisions. 
Models  and  model-generated  information  influences  decision  maker  behavior  in  various  ways. 
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Together,  humans  and  models  exhibit  dynamic  behavior.  Understanding  human-model 
interaction  is  predicated  on  taking  this  endogenous  point  of  view. 


Model-Centric  Decision  Making 

Understanding  the  current  state  of  the  practice  is  an  important  step  in  model-centric  enterprise 
transformation.  The  findings  in  our  recent  interview-based  study  involving  30  experts  led  to 
insights,  along  with  information  from  literature  and  secondary  source  studies,  have  informed 
the  derivation  of  heuristics  for  human-model  interaction.  Findings  in  the  study  include  the 
areas  below,  with  two  of  these  areas  briefly  described.  A  more  detailed  discussion  of  the 
findings  is  presented  in  a  recent  paper  (German  &  Rhodes,  2017)  and  in  the  IMCSE  Phase  4 
report  (Rhodes  &  Ross,  2017). 

•  Three  actor  decision  flow 

•  Understanding  of  assumptions  and  uncertainty 

•  Technological  and  social  factors  influencing  trust 

•  Importance  of  documentation  and  model  pedigree 

•  Using  models  as  primary  versus  supplementary 

•  Model  transparency  and  model  trust 

•  Factors  and  biases  limiting  model-centric  decisions 

•  Modes  of  interaction  with  models 


Models  and  Trust 

A  major  theme  arising  from  an  interview-based  study  of  current  model-centric  decision  making 
showed  a  pattern  described  as  a  three  actor  decision  flow  (German  &  Rhodes,  2017).  The  flow 
begins  with  the  modelers  and  ends  with  the  ultimate  decision  maker.  In  the  middle  of  the  flow 
is  the  "through"  person  who  is  a  trusted  agent  of  the  decision  maker.  The  data  suggests  that  as 
actors  move  further  along  the  flow  of  information  and  have  less  time  and  ability  to  personally 
investigate  a  model  and  build  their  own  trust  in  the  model,  their  trust  instead  shifts  more  onto 
their  people  to  investigate  the  model  for  them.  Trust  for  the  ultimate  decision-maker  is 
"implicitly  on  the  models,  but  explicitly  on  the  people."  A  combination  of  technical  and  social 
factors  is  involved,  as  shown  in  Figure  3. 


Report  No.  SERC- 2017- TR- 104 


6 


Date  February  28,  2018 


Model  code 
Referent  data 


Uncertainty 

Documentation 


Personal  relationships 
Experience  level 
Originator  of  model 
Expert  opinions 


Figure  3.  Three  Actor  Decision  Flow 

A  model  does  not  provide  value  by  existing.  Rather,  it  provides  value  by  producing  information 
that  can  be  used  by  individuals  and  decision-makers  to  better  understand  a  specific  problem. 
Often,  information  flows  from  initial  actors  that  directly  interface  with  a  model  before  flowing 
through  other  individuals  within  the  decision-making  process  and  ultimately  to  final  decision¬ 
makers.  All  actors  in  the  model-generated  flow  of  information  must  have  a  well-calibrated 
understanding  of  the  model  if  the  information  is  to  be  effectively  understood. 


Various  Factors  Limiting  Effective  Model-Centric  Decisions 

The  model-centric  decision-making  study  revealed  factors  perceived  as  limiting  effective  model¬ 
centric  decisions.  While  technical  (model-related)  factors  are  cited,  the  majority  of  limitations 
that  the  expert  interviewees  raised  related  to  human  (social  and  cultural)  factors.  Figure  4 
shows  selected  examples  of  factors  cited  in  the  interviews. 


MODEL 


HUMAN 


Data  availability 


Talent  of  people  Skill  level  Generational  differences 


Data  quality 

Model  complexity 

Inadequate  methods 

Lack  of  transparency  and 
model  documentation 

Interactivity  with  models 


Inertia  to  change  Ability  to  socialize 

_  ....  models 

Communication  barriers 


Changing  preferences  of 
decision-makers 


Lack  of  trust/fear  of  the 
unknown 


Lack  of  understanding 

Unwillingness  to  share 

models  or  information  Educated  leadership 


Organizational 

differences 


Time  and  money 
Team  agreement 


Lack  of  desire  to 
understand 

Bad  past  experiences 


Figure  4  Factors  Limiting  Effective  Model-Centric  Decisions 
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Human-Model  Interaction  Heuristics 


The  research  has  informed  an  initial  body  of  heuristics  that  encapsulate  strategies  that  can  be 
used  in  practice  for  several  purposes.  Sources  of  information  include  the  IMCSE  Pathfinder 
Workshop  (Rhodes  &  Ross,  2015),  theory  and  secondary  sources,  analogy  cases  (German  & 
Rhodes,  2016)  and  the  expert-interview  study  (German  &  Rhodes,  2017). 

Multiple  purposes  are  proposed  for  the  heuristics.  These  include  educating  novices,  conducting 
expert  forums,  informing  policy  and  practice,  and  conducting  program  team  launches. 
Heuristics  can  be  thought  of  as  prescriptive  theories  that  offer  "suggestions  to  facilitate  more 
extensive  search  for  useful  possibilities  and  evidence"  (Baron,  1988).  Maier  &  Rechtin  (2000) 
define  heuristics  as  "sophisticated  abstractions  of  lessons  learned  from  experience."  Such 
condensed  and  codified  abstractions  of  practical  experience  should  be  thought  of  as  tools  to 
assist  individuals  in  design  and  decision-making.  There  may  be  hundreds  of  potential  heuristics 
available  to  a  decision-maker,  "but  only  a  few  are  needed  at  any  one  time  and  for  a  specific  job 
at  hand"  (Maier  &  Rechtin,  2000).  Heuristics  are  "guides  along  the  way,"  but  also  "must  be  used 
with  judgment".  There  is  an  art  to  this  discernment,  not  in  evaluating  the  wisdom  of  heuristics 
themselves,  but  "in  the  wisdom  of  knowing  which  heuristics  apply"  to  a  given  problem  (Maier  & 
Rechtin,  2000). 

Recent  research  studies  have  shown  the  benefit  of  using  heuristics  to  assist  novice  engineers  in 
early  design  activities  (Yilmaz  et  al.  2016,  Murphy  et  al.,  2017).  Using  heuristics  for  launching  an 
engineering  effort  is  supported  by  Maier  &  Rechtin  (2000);  the  authors  state  they  are  "trusted, 
nonanalytic  guidelines"  and  can  be  used  "as  aids  in  decision  making,  value  judgements  and 
assessments". 

The  heuristics  generated  in  this  research  have  been  initially  validated  through  several  activities, 
and  subsequently  refined.  Further  work  is  ongoing  to  fine-tune  the  heuristics,  as  well  as  expand 
the  set  based  on  additional  research.  Ten  heuristics  are  presented  and  explained  below. 


Humans  should  not  be  forced  to  adapt  to  models,  rather,  models  should  be  designed  for  humans. 

Evolving  technology  enables  more  complex  and  capable  models  but  may  not  increase 
effectiveness  if  humans  are  not  appropriately  considered.  Humans  have  cognitive  and 
perceptual  limitations  that  limit  the  amount  and  type  of  information  they  can  effectively 
comprehend  and  use  to  make  decisions.  Designing  for  humans  requires  understanding  their 
capabilities  and  limitations  so  that  the  model  intelligence  can  extend  the  overall  system 
intelligence.  Model  designers  must  consider  the  users  and  how  the  model  will  influence  their 
behavior  and  decisions. 
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Model  developers,  model  users,  and  organizations  share  responsibility  for  preventing  improper 

MODEL  USAGE. 

Model  developers  have  the  responsibility  to  make  capabilities  and  limitations  of  a  model  salient 
and  understandable,  to  enable  decisions  on  proper  use.  Model  users  have  the  responsibility  to 
obtain  and  use  this  information  in  determining  use/reuse  of  a  model,  as  well  as  documenting 
this  decision  with  assumptions.  Organizations  have  the  responsibility  to  prevent  the  improper 
use  of  models  in  general,  through  effective  policies,  practices  and  controls. 


Models  do  not  have  agency  --  the  ultimate  responsibility  for  decisions  must  be  upon  humans. 

Increasingly  complex  and  automated  modeling  environments  may  make  it  easier  to  attribute 
agency  and  responsibility  to  models.  The  ultimate  decision-making  authorities  are  people  and 
blame  cannot  be  placed  upon  models  for  poor  decisions.  Individuals  should  be  aware  of  the 
potential  for  improperly  diffusing  responsibilities  for  decisions  upon  models,  and  policies  should 
clearly  establish  the  responsibilities  for  which  individuals  are  held  accountable. 


Before  any  model  can  be  useful,  its  capabilities  and  limitations  must  be  revealed  and  understood. 

All  models  are  inherently  abstractions  of  reality  that  contain  assumptions  about  the  modeled 
system,  and  these  assumptions  limit  the  applicability  of  where  and  how  the  model  can  be  used. 
Numerous  empirical  examples  show  that  poorly  understood  and  applied  models  have  led  to 
programmatic  challenges  and  failures.  An  improper  understanding  of  a  model's  limitations  may 
lead  to  decision-makers  inappropriately  applying  it  beyond  its  limitations.  Conversely,  if  a 
model's  capabilities  are  not  fully  understood,  decision-makers  may  not  use  the  model  to  its 
fullest  potential. 


Models  are  created  for  specific  reasons  and  contexts  that  fundamentally  bound  a  model's 

APPLICABILITY. 

A  model  may  be  insightful  and  valuable  within  one  problem  context,  but  the  assumptions  built 
into  the  model  may  not  be  valid  within  some  other  contexts.  Evaluating  a  model's  applicability 
should  not  just  consider  whether  it  has  been  verified  and  validated,  but  also  under  what 
conditions  and  in  what  contexts.  Using  a  model  outside  of  its  inherent  bounds  may  lead  to 
model  results  that  are  inappropriate  for  the  problem  under  consideration. 


Model  documentation  should  make  the  assumptions  and  limitations  of  a  model  explicit;  otherwise 

IT  WILL  NOT  BE  USABLE  BY  ANYONE  OTHER  THAN  THE  ORIGINATOR. 

Model  developers  carry  the  most  intimate  understanding  of  a  model's  assumptions  and 
limitations.  If  decision-makers  are  those  other  than  the  modelers,  however,  assumptions  and 
limitations  must  be  clearly  documented  so  that  others  might  calibrate  an  appropriate  personal 
understanding  of  the  model.  Documentation  should  not  only  capture  the  assumptions  built 
within  the  model,  but  the  assumptions  made  about  the  model  itself.  Conceptual  "white  board" 
artifacts  created  early  in  a  model's  development  can  offer  insight  into  decisions  made  about  a 
model,  including  what  problem  contexts  the  model  is  designed  for.  With  this  line  of  thinking. 
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both  decisions  made  regarding  assumptions  within  the  model's  code  and  the  decisions  made 
before  the  model  was  even  designed  should  be  documented.  If  these  assumptions  and 
limitations  are  not  documented  and  accessible,  users  and  decision-makers  will  not  be  able  to 
appropriately  calibrate  their  understanding  and  trust  of  the  model,  which  makes  it  unusable. 


Model  trust  is  a  sociotechnical  construct,  you  must  examine  both  technical  and  social  factors  to 

UNDERSTAND  HOW  INDIVIDUALS  DEVELOP  TRUST  IN  MODELS. 

Individuals  within  the  model-centric  decision-making  process  rely  upon  various  technical  and 
social  factors  to  develop  trust  in  a  model.  Technological  factors  include  technical  information 
about  a  model,  such  as  its  transparency,  uncertainty,  and  input  data.  Social  factors  include  the 
people,  organizations,  and  relationships  that  shape  one's  trust  of  a  model.  These  social  aspects 
could  include  factors  such  as  the  credibility  of  the  people  or  organization  developing  the  model, 
reliability  of  the  relationship  with  individuals  recommending  a  model,  or  word  of  mouth  within 
a  community  concerning  a  model's  performance.  Different  factors  may  play  greater  or  lesser 
roles  in  developing  an  individual's  model  trust  process;  therefore,  these  factors  should  be 
understood  to  facilitate  appropriate  calibration  of  individual  trust. 


Models  should  have  the  capability  to  be  as  transparent  as  possible;  however,  not  every  user  desires 

FULL  TRANSPARENCY.  TRANSPARENCY  SHOULD  BE  TAILORED  TO  THE  NEEDS  OF  THE  SPECIFIC  INDIVIDUAL 

UNDER  CONSIDERATION. 

Transparency  involves  how  clearly  one  might  assess  a  model's  functions  and  understand  how 
and  why  it  operates  as  it  does.  This  allows  individuals  to  determine  if  the  operation  is 
appropriate  for  a  decision  at  hand.  Full  transparency  would  involve  having  complete  access  to  a 
model's  code  and  documentation  of  the  assumptions  built  within  it.  While  there  should  always 
be  the  opportunity  for  full  transparency,  individuals  may  desire  different  levels  of  transparency. 
For  example,  high-level  decision-makers  may  only  desire  transparency  concerning  high-level 
model  assumptions  because  they  lack  the  time  or  training  to  effectively  investigate  a  model's 
code.  Too  much  transparency  could  cause  an  information  overload  that  obscures  the  relevant 
information.  Conversely,  others  may  desire  the  ability  to  be  intimately  acquainted  with  a 
model's  workings.  Transparency,  therefore,  should  always  be  present,  but  should  be  tailored  to 
the  needs  of  the  specific  individual  under  consideration. 


Appropriate  model  trust  calibration  includes  sharing  of  mental  models  between  actors  in  a 

DECISION-MAKING  FLOW. 

A  model  is  the  explicit  representation  of  the  model  developers'  mental  model  of  a  situation. 
This  mental  model  contains  an  implicit  understanding  of  the  capabilities  and  limitations  of  the 
model.  The  other  individuals  who  interact  with  the  model  should  calibrate  their  own  mental 
models  to  that  of  the  model  developers,  which  includes  developing  an  accurate  understanding 
of  the  model's  capabilities  and  limitations.  Improperly  calibrated  mental  models  lead  to 
inappropriate  decisions  on  an  individual  basis,  but  as  modeling  is  a  sociotechnical  process, 
improper  mental  models  may  also  influence  and  hinder  proper  mental  model  development  in 
other  individuals  within  the  flow  of  information. 
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Increasing  the  speed  of  decision-making  implies  a  decrease  in  time  spent  analyzing  a  problem,  which 

IN  TURN  INCREASES  THE  CHANCE  OF  BIASED  JUDGMENTS. 

Humans  excel  at  pattern  recognition  and  efficiently  jumping  to  conclusions  in  the  presence  of 
limited  information.  These  intuitive  capabilities  prove  effective  for  many  day-to-day  activities, 
but  they  can  also  lead  to  systematic  biases  in  certain  contexts.  Model-centric  environments 
may  seek  to  provide  a  more  intuitive  environment  that  users  and  decision-makers  can  directly 
interact  with  to  build  intuition  and  speed  decision-making.  However,  these  environments  may 
make  biases  more  likely  by  allowing  users  to  quickly  match  patterns  and  move  forward  with 
incomplete,  and  potentially  biased,  understandings  of  a  situation.  Complex  problems  may 
require  focused  attention  and  analysis  that  take  time  to  fully  understand  in  order  to  develop  an 
accurate  mental  model  of  the  situation.  While  faster  decisions  are  desired  if  they  are  effective, 
the  speed  itself  may  set  people  up  for  failure  by  encouraging  them  to  rely  upon  their  fast  and 
intuitive,  yet  bias-susceptible,  judgment,  rather  than  the  more  cognitively  demanding  rational 
and  analytical  thought  processes. 


Heuristics  as  Practical  Enablers 

There  are  four  application  areas  that  have  been  proposed  for  the  heuristics  as  practical 
enablers  for  model-centric  engineering  practice.  These  are:  educating  novices,  conducting 
expert  forums,  conducting  program  team  launches,  and  informing  policy  and  practice. 

Educating  Novices 

A  first  application  area  is  using  the  heuristics  as  part  of  basic  education  courses  for  model- 
based  engineering,  aimed  at  novices  (i.e.,  individuals  with  relatively  little  engineering 
experience).  Recent  research  studies  show  benefit  of  using  heuristics  to  assist  novice  engineers 
in  early  design  activities  (Yilmaz  et  al.,  2016;  Murphy,  et  al.,  2017). 

Beginning  engineering  students  may  learn  to  use  model-based  toolsets  quickly  (having  grown 
up  using  software  applications);  however,  they  lack  the  foundational  knowledge  behind  the  use 
of  models  in  engineering  decisions.  By  incorporating  the  heuristics  into  education  modules  and 
classroom  exercises,  they  serve  to  amplify  key  learning  points.  Given  that  heuristics  are  concise 
statements,  they  have  potential  to  make  key  points  stick  in  the  mind.  Accordingly,  they  are 
useful  for  triggering  class  discussion.  As  an  example,  a  teaching  lesson  on  the  importance  of 
understanding  context  and  assumptions  could  employ  the  following  heuristic  to  initiate  a 
discussion:  Models  are  created  for  specific  reasons  and  context,  and  those  assumptions 
fundamentally  bound  a  model's  applicability. 

A  discussion  on  this  heuristic  may  revolve  around  a  model  as  insightful  and  valuable  within  one 
problem  context,  but  the  assumptions  built  into  the  model  may  not  be  valid  within  some  other 
context.  A  key  point  is  that  evaluating  a  model's  applicability  should  not  just  consider  whether 
it  has  been  validated,  but  in  what  contexts  the  model  has  been  validated.  Another  useful 
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discussion  point  is  that  using  a  model  outside  of  its  inherent  bounds  leads  to  model  results  that 
are  inappropriate  for  the  problem  under  consideration.  Specific  case  examples  could  be  used  to 
reinforce  these  points.  Another  use  of  a  heuristic  in  education  would  be  an  assignment  to  write 
a  short  paper  to  elaborate  on  the  heuristic. 

Conducting  Expert  Forums 

Enterprises  undergoing  transformation  to  model-centric  engineering  practice  experience  the 
challenges  of  re-orienting  the  workforce  from  traditional  engineering  to  model-centric 
engineer.  The  more  senior  engineers  with  a  long  history  of  performing  specification-based 
engineering  are  faced  with  doing  things  differently  using  modeling  technologies  and  languages 
that  are  unfamiliar.  These  engineers  can  be  challenged  with  making  decisions  with  model¬ 
generated  information  in  formats  they  perceive  as  not  naturally  comprehensible.  This 
resistance  to  adoption  of  model-based  engineering  is  cited  as  a  challenge  faced  by  leadership 
today  (IAWG,  2017). 

A  factor  that  has  major  impact  on  engineering  outcomes  is  the  degree  of  trust  in  models.  It  is 
not  that  models  are  new  for  the  expert  workforce,  but  the  changed  manner  and  degree  to 
which  models  are  used  in  performing  engineering  tasks.  In  the  future,  models  will  be  the  "single 
source  of  truth"  (Blackburn  et  al.,  2015).  The  digital  thread  brings  together  models  and 
communication  mechanisms  in  a  way  in  which  they  may  not  be  aligned  with  the  organization 
and  traditional  ways  of  doing  things  (West  &  Pyster,  2015). 

There  is  potential  benefit  in  the  use  of  expert  forums  to  gather  together  members  of  the  expert 
workforce  for  dialogue  on  the  changing  landscape.  This  provides  an  opportunity  to  raise 
concerns,  and  through  discussion  promote  understanding  of  the  new  landscape  for  performing 
engineering.  Heuristics  can  be  used  in  this  way  to  trigger  discussion  among  senior  practitioners. 
The  heuristics  represent  insights  from  an  expert-level  practitioner  for  example:  Increasing  the 
speed  of  decision-making  implies  a  decrease  in  time  spent  analyzing  a  problem,  which  in  turn 
increases  chance  of  biases  judgement.  This  heuristic  could  prompt  meaningful  discussion  on 
the  perceived  value  and  risks  of  emphasizing  efficiencies  of  models. 

Conducting  Program  Team  Launches 

A  third  application  for  the  heuristics  is  in  conducting  team  launches  at  the  start  of  a  program  or 
major  phase  on  an  ongoing  program.  A  program  team  launch  brings  together  the  members  of  a 
team  in  a  congenial  forum  to  plan  activities,  assign  responsibilities,  and  set  team  norms. 
Beyond  practical  planning,  a  team  launch  is  meant  to  enhance  trust  and  build  collaborative 
spirit  among  team  members.  Using  heuristics  for  launching  a  model-centric  program  is 
supported  by  Maier  &  Rechtin,  who  state  they  are  "trusted,  nonanalytic  guidelines"  and  can  be 
used  "as  aids  in  decision  making,  value  judgements  and  assessments".  (Maier  &  Rechtin,  2009). 
Heuristics  have  potential  to  serve  as  a  basis  for  an  activity  to  set  norms  and  team  practices  for 
the  project.  A  common  approach  on  a  new  model-centric  program  is  to  send  all  the  engineers 
through  tool  training.  Younger  engineers  may  quickly  learn  to  use  modeling  tools,  but  lack  the 
experiential  knowledge  of  engineering  of  products  and  systems.  The  members  with  years  of 
experience,  on  the  other  hand,  may  find  use  of  modeling  software  tools  to  be  non-intuitive  to 
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the  point  where  they  spend  more  time  on  tool  mechanics  and  less  on  decision  making.  Any 
discomfort  and  distrust  of  models  and  modeling  toolsets  can  have  negative  impacts  on  the 
engineering  effort. 

By  using  heuristics  as  "boundary  objects",  there  is  an  opportunity  for  team  dialogue  on  how  the 
project  wants  to  employ  models  in  context  of  the  program  and  the  competencies  of  the  team 
members.  An  example  of  a  useful  heuristic  for  such  a  discussion  is:  Models  do  not  have  agency 
-  the  ultimate  responsibility  for  decisions  must  be  upon  the  humans.  A  team  discussion  on 
this  heuristic  could  prompt  discussion  on  the  proper  use  of  models  in  context  of  the  specific 
program  objectives,  and  how  model-generated  information  is  to  be  used  by  various  team 
members  on  the  project.  Additionally,  decision  authorities  and  standard  practices  for 
documenting  model-related  decisions  can  be  discussed  and  established.  This  heuristic  can  be 
used  to  invoke  a  discussion  on  what  models  will  be  used  on  the  program,  what  decisions  are 
expected  to  be  informed  by  the  model,  and  how  model  information  is  to  be  validated  using 
expert  review.  Using  a  heuristic  as  boundary  object  encourages  more  open  and  non- 
confrontational  discussion. 

Informing  Policy  and  Practice 

A  fourth  application  of  the  heuristics  is  to  support  policy,  practices  and  guidelines.  As  the 
heuristics  are  essentially  encapsulations  of  expert  strategies,  they  are  useful  for  framing  key 
points  that  can  be  elaborated  in  policy  and  practices  that  an  organization  develops.  An 
example  heuristic  related  to  designing  models  for  human  use  is:  Humans  should  not  be  forced 
to  adapt  to  models,  rather  models  should  be  designed  for  humans.  It  suggests  the  need  to 
have  practices  that  identify  capabilities  to  understand  a  model  and  identify  mechanisms  to 
ensure  models  generate  information  in  a  manner  that  it  can  be  cognitively  and  perceptually 
processed  by  the  model  user  (e.g.,  visualization  enabling  zooming  to  understand  patterns  in  the 
data).  Many  organizations  are  revising  engineering  practices  as  part  of  the  transformation  to  a 
model-centric  enterprise.  This  heuristic  could,  for  example,  prompt  a  discussion  on  how  to 
include  the  essence  of  the  heuristic  in  evaluation  of  new  models. 


Discussion  and  Future  Directions 

One  limitation  of  the  interview-based  study  was  that  it  had  a  relatively  small  population  size  of 
thirty  experts,  with  the  majority  coming  from  the  aerospace  and  defense  sector.  Further,  the 
subjects  were  primarily  "through"  actors.  Additional  study  is  necessary  for  more  generalizable 
results,  with  inclusion  of  additional  types  of  human  actors  in  model-centric  engineering  and 
additional  sectors. 

Limited  validation  of  the  heuristics  has  been  performed  in  the  research  to  date.  In  the  next 
phase,  research  will  include  iterative  evaluation  and  refinement  of  heuristics  from  a  larger  pool 
of  evaluators,  in  order  to  assess  the  heuristics  as  to  their  applicability  and  enhance  potential 
acceptance.  Endsley,  et  al.  (2017)  found  that  "a  process  of  iteration  through  evaluation  is 
effective  for  the  creation  of  robust  sets  of  heuristics  for  designers  and  evaluators". 
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Planned  future  research  will  use  the  validated  heuristics  (guiding  principles)  as  the  basis  for  two 
pilot  applications:  (1)  a  teaching  module  for  use  in  an  undergraduate  engineering  leadership 
program  and  (2)  a  segment  for  use  in  a  model-based  engineering  program  team  launch. 


Summary 

The  research  has  confirmed  the  need  for  further  investigation  on  myriad  topics  related  to 
human-model  interaction.  One  is  the  need  to  formally  capture  the  patterns  of  why,  when  and 
how  various  stakeholders  use  models.  Studies  and  experiments  are  needed  to  better 
understand  the  most  effective  means  and  conditions  for  human  interaction  with  models. 
Future  plans  include  further  investigation  of  knowledge  from  related  fields.  An  important  topic 
for  future  research  is  use  of  augmented  intelligence  in  model-informed  decisions.  This  includes 
understanding  engineering  tasks  and  decisions  that  necessitate  human  actors  over  automation, 
in  context  of  the  model-centric  engineering  effort. 
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Interactive  Epoch-Era  Analysis 


The  Interactive  Epoch-Era  Analysis  (IEEA)  framework  has  been  developed  by  adapting  and 
extending  the  existing  Epoch-Era  Analysis  (EEA)  method  through  interaction  and  visualizations 
and  associated  infrastructure  advancements.  IEEA  is  proposed  as  an  iterative  framework  for 
concept  exploration  that  provides  a  means  of  applying  EEA  constructs  while  controlling  growth 
in  data  scale  and  dimensionality.  Further,  IEEA  leverages  interactive  visualization  because  prior 
visual  analytics  research  has  demonstrated  that  when  performing  exploratory  analysis,  like 
early-phase  system  concept  selection,  an  analyst  can  gain  deeper  understanding  of  data  which 
can  lead  to  improved  decision-making. 

In  Phase  3,  the  framework  and  supporting  tools  were  applied  to  a  multi-mission  on-orbit 
servicing  vehicle  (Curry  &  Ross,  2016),  demonstrating  key  concepts  and  prototype  interactive 
visualizations,  focusing  on  opportunities  to  improve  the  uncertainty  analysis,  ease  of  use,  data 
scaling,  visualization  techniques,  and  overall  analysis  approach.  In  Phases  4  and  5,  the  research 
was  applied  to  a  commercial  ship  case  to  further  test  the  framework  and  refine  visualization 
prototypes  (Curry  et  al.,  2017).  Discussion  of  the  interactive  prototypes  can  be  found  in  the 
IMCSE  prior  phase  report  (Rhodes  &  Ross,  2016),  with  detailed  implementation  discussion  in  a 
recent  doctoral  thesis  (Curry,  2017).  An  impact  assessment  experiment,  initiated  in  Phase  4, 
aimed  to  decouple  and  evaluate  the  impacts  of  visualization  and  interaction  on  human 
performance.  The  experiment  was  completed  in  Phase  5. 


Background 

While  the  transformation  of  engineering  to  a  model-centric  approach  does  involve  some  new 
frameworks,  methods  and  tools,  it  does  not  involve  abandoning  those  that  are  in  use  in 
engineering  practice.  Rather,  there  is  a  need  to  adapt  the  proven  frameworks  and  methods  for 
model-centric  engineering.  This  includes  the  need  to  make  these  more  interactive  and  extend 
them  as  needed  to  accommodate  larger-scale  problems  and  data  sets.  Similarly,  there  is  a 
desire  to  make  use  of  the  latest  research  and  approaches  from  data  science  and  visual  analytics 
in  the  extension  of  these  existing  frameworks,  methods  and  tools.  IMCSE  has  explored  how  an 
existing  framework,  Epoch-Era  Analysis  (EEA),  can  be  adapted  and  extended  in  such  a  manner. 
Epoch-Era  Analysis  (EEA),  supports  narrative  and  computational  scenario  planning  and  lifecycle 
uncertainty  analysis  for  both  short  run  and  long  run  futures.  EEA  is  designed  to  clarify  the 
effects  of  changing  contexts  over  time  on  the  perceived  value  of  a  system  in  a  structured  way 
(Ross  &  Rhodes,  2008).  The  base  unit  of  time  in  EEA  is  the  epoch,  which  is  defined  as  a  time 
period  of  fixed  needs  and  context  in  which  the  system  exists.  This  approach  provides  an 
intuitive  basis  upon  which  to  perform  analysis  of  value  delivery  over  time  for  systems  under  the 
effects  of  changing  circumstances  and  operating  conditions,  an  important  step  to  take  when 
evaluating  large-scale  engineering  systems  with  long  lifecycles. 

EEA  has  been  applied  in  numerous  case  studies  and  in  limited  use  in  performing  selected 
studies  on  real  program  studies.  Some  limitations  in  its  application  have  been  due  to  the  large 
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scale  of  data  sets  generated  in  the  engineering  of  complex  systems.  Because  of  the  complex 
data  that  must  be  analyzed  when  extending  EEA  to  large-scale  problems,  issues  with  cognition 
are  introduced  that  may  hamper  decision-making.  This  motivates  the  need  for  extensions  to 
EEA  methods  that  overcome  the  computational  and  human  cognition  issues  that  may  arise.  It  is 
hypothesized  (Curry  &  Ross,  2015)  that  augmenting  the  traditional  EEA  approach  with  new 
analytic  and  interactive  techniques  will  fundamentally  enable  new  capabilities  and  insights  to 
be  derived  from  EEA,  resulting  in  superior  dynamic  strategies  for  resilient  systems. 

The  Interactive  Epoch-Era  Analysis  (IEEA)  Framework  has  resulted  from  development,  testing 
and  refinement  activities  over  the  past  several  phases  of  research. 


IEEA  Framework 

Interactive  Epoch-Era  Analysis  (IEEA)  leverages  human-in-the-loop  (HIL)  interaction  to  manage 
challenges  associated  with  the  large  amounts  of  data  potentially  generated  in  a  study,  as  well 
as  to  improve  sense-making  of  the  results  (Curry  &  Ross,  2015).  Allowing  the  structured 
evaluation  and  visualization  of  many  design  alternatives  across  many  different  futures  and 
potential  lifecycle  paths  enables  the  design  of  systems  that  can  deliver  sustained  value  under 
uncertainty.  Iteration  is  necessary  because  the  analysis  is  inherently  exploratory  in  nature.  HIL 
interaction  is  necessary  because  the  problem  is  not  strictly  deterministic  or  necessarily 
intended  as  a  reliable  prediction  of  system  performance  or  future  events.  Often,  there  is  both 
uncertainty  and  the  potential  for  errors  in  assumptions  or  model  implementation.  This 
necessitates  human  judgment  to  make  sense  of  the  data;  therefore,  this  is  not  by  its  nature  a 
problem  that  can  be  handed  over  completely  to  an  automated  optimization  algorithm,  though 
some  level  of  automated  analysis  could  be  beneficial  as  an  aid  to  the  user.  Enabling  users  to 
interact  with  their  data  through  visual  interfaces  of  this  type  is  an  area  of  active  research  (Heer 
&  Agrawala,  2007,  Heer  &  Shneiderman,  2012). 

Description  of  IEEA  Framework  Modules 

The  purpose  of  IEEA  is  to  "guide  the...  practitioner  through  the  steps  of  determining  how  a 
system  will  deliver  value,  brainstorming  solution  concepts,  identifying  variances  in  contexts  and 
needs  (epochs)  that  may  alter  the  perceived  value  delivered  by  the  system  concepts,  evaluating 
key  system  trade-offs  across  varying  epochs  (eras)  to  be  encountered  by  the  system,  and  lastly 
developing  strategies  for  how  a  designer  might  develop  and  transition  a  particular  system 
concept  through  and  in  response  to  these  varying  epochs".  To  that  end,  the  IEEA  framework,  as 
shown  in  Figure  5,  is  characterized  by  10  individual  processes  that  can  be  abstracted  into  five 
main  modules: 

1.  Elicitation  of  relevant  epoch  and  design  variables  (often  through  interview), 

2.  Generation/Sampling  of  all  epochs,  eras  and  design  tradespaces  (often  including 
enumeration)  in  which  to  evaluate  design  choices, 

3.  Evaluation  of  designs  in  sampled  subset(s)  of  epochs  and  eras 

4.  Analyses  of  design  choices  in  the  previously  evaluated  epochs  and  eras,  and  finally 

5.  Decisions  of  final  designs  based  on  iterative  evidence  from  previous  modules. 
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Figure  5.  Interactive  Epoch-Era  Analysis  Framework 


While  the  sequence  of  these  modules  flows  logically,  IEEA  is  intended  to  be  an  iterative  process 
where  users  can  go  back  and  change  responses  within  earlier  modules  at  any  point  to  reflect 
what  they  have  learned  from  later  ones.  The  five  modules  are  composed  of  the  10  processes, 
but  depending  on  the  nature  of  the  study  and  the  type  and  fidelity  of  information  available  to 
the  analyst,  it  is  not  strictly  required  that  each  process  step  be  employed.  From  a  computing 
infrastructure  perspective,  many  of  the  techniques  discussed  in  Curry  et  al.  (2015)  can  be 
applied  to  augment  and  facilitate  a  practical  implementation  of  the  workflow.  For  example, 
online  analytical  processing  (OLAP)  techniques  may  be  applied  to  improve  data  handling,  and 
search  algorithms  may  improve  our  ability  to  offer  more  informed  recommendations  to 
decision-makers  during  the  epoch-era  analysis.  Similarly,  enhanced  human  interaction 
techniques  and  visualizations  may  aid  in  the  analyses  of  the  vast  amounts  of  information 
required  to  reach  an  informed  decision. 


Demonstration  of  Interactive  Visualization 

The  IEEA  framework  research  furthered  understanding  of  the  use  of  interactive  visualizations  in 
complex  engineering  system  studies.  The  visualization  prototypes  were  developed  based  on 
prior  relevant  research,  and  with  continuous  feedback  using  informal  review  from  subject 
matter  experts. 

Prior  Relevant  Research 

Effective  analysis  of  EEA  generated  data  to  derive  valuable  insights  enables  analysts  to  make 
informed  decisions  and  design  successful  strategies  for  value  sustainment.  Techniques  from 
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visual  analytics  are  central  to  IEEA,  as  evidenced  by  their  successful  application  in  other 
domains  to  solve  real-world  problems.  Visual  analytics  applications  are  beneficial  for  addressing 
problems  whose  size,  complexity,  and  need  for  closely  coupled  human  and  machine  analysis 
may  make  them  otherwise  intractable  (e.g.,  Chang  et  al.,  2007).  An  example  is  the  creation  of  a 
visual  analytics  application  for  detecting  financial  wire  fraud  (Yang  et  al.,  2008).  In  banking,  the 
nature  of  fraud  is  ill-defined  and  constantly  evolving,  much  like  the  nature  of  value  among 
different  decision  makers  in  system  design.  A  similar  comparison  can  be  made  with  recent 
visual  analytics  applications  in  the  field  of  healthcare  that  must  also  frequently  contend  with  ill- 
defined  problems.  The  same  data,  explored  by  different  medical  professionals  can  lead  to 
different  decisions  for  a  variety  of  reasons  including  risk  aversion,  available  resources,  or 
experience-level  (Caban  &  Gotz,  2015).  Additional  examples  can  be  found  in  such  diverse 
domains  as  infrastructure  maintenance  (Wang  et  al.,  2010)  and  understanding  the  nature  of 
global  terrorism  (Chang,  2008).  Clearly  many  other  large-scale  problems  have  seen  significant 
benefits  by  coupling  skills  of  subject  matter  experts  with  well-developed  visual  analytics 
applications  that  help  them  consider  and  comprehend  complex  data. 

Visual  analytics  extends  beyond  traditional  scientific  visualization  and  focuses  on  extracting 
insights  from  data  using  interactive  visual  interfaces  (Wong  &  Thomas,  2004).  Interactive 
visualizations  are  used  to  integrate  a  user's  knowledge  and  inference  capability  with  numerical 
and  algorithmic  data  analysis  processes.  Thomas  &  Cook  (2005)  presented  the  first  widely 
accepted  roadmap  for  visual  analytics  research  in  their  seminal  book.  As  they  described,  the 
research  agenda  in  this  field  seeks  to  develop  "the  science  of  analytical  reasoning  facilitated  by 
interactive  visual  interfaces".  Keim  et  al.  (2010)  updated  the  roadmap  and  provided  the 
following  revised  definition  of  visual  analytics:  "Visual  analytics  combines  automated  analysis 
with  interactive  visualizations  for  effective  understanding,  reasoning  and  decision  making  on 
the  basis  of  a  very  large  and  complex  dataset".  Icke  (Icke  &  Sklar,  April  16,  2009),  Keim  (2010), 
Keim  et  al.  (2010)  and  Sun  et  al.  (2013)  provide  good  overviews  of  the  state  of  current  research 
on  visual  analytics.  While  there  are  obviously  many  different  areas  of  investigation  ongoing  in 
visual  analytics  research  there  are  three  particular  areas  that  are  highly  relevant  to  the  current 
effort  for  IEEA:  (1)  methods  to  facilitate  user  interaction  with  data;  (2)  specific  types  of 
visualizations  and  frameworks;  and  (3)  data  reduction  and  handling  of  large  amounts  of  data. 

Visual  analytics  research  on  methods  for  interacting  with  data  to  extract  useful  insights 
provides  some  guidance  to  the  current  research  on  IEEA.  As  a  visual  metaphor  for  describing 
how  data  interaction  should  be  applied  to  gain  insights,  Shneiderman  et  al.  (1996)  proposed  the 
famous  information  seeking  mantra:  "Overview  first,  zoom/filter,  details  on  demand". 
Highlighting  the  need  for  coupling  numerical  and  algorithmic  data  analysis  with  interactive 
visual  interfaces  to  gain  insights  during  visual  data  exploration  Keim  et  al.  (2006)  later  extended 
the  mantra.  The  revised  version  called  the  visual  analysis  mantra  was:  "Analyze  first,  show  the 
important,  zoom/filter,  analyze  further,  details  on  demand".  Prototype  applications  for  IEEA 
follow  the  visual  analysis  mantra  as  a  guideline. 

The  visual  analytics  process  introduced  by  Keim  et  al.  (2010),  as  shown  in  Figure  6,  is 
characterized  through  interaction  between  data,  visualizations,  models  of  the  data,  and  the 


Report  No.  SERC- 2017- TR- 104 


18 


Date  February  28,  2018 


users,  in  order  to  discover  knowledge.  The  process,  shown,  begins  by  transforming  the  data 
(e.g.  filtering,  sampling,  cleaning)  so  that  it  can  be  analyzed  further.  Next,  visual  or  algorithmic 
analysis  methods  are  applied  to  explore  the  data.  When  visual  data  exploration  is  used,  users 
directly  interact  with  the  visual  interface  to  analyze  and  explore  the  data.  When  automatic 
analysis  methods  are  applied,  approaches  such  as  regression  modeling  or  pattern  matching  are 
used  to  estimate  models  for  characterizing  the  data. 

Visual  Data  Exploration 


o  User  interaction 


Figure  6.  Keim  et  al.  (2010)  on  Solving  Problems  with  Visual  Analytics 

Numerous  frameworks  for  developing  interactive  visualization  applications  exist.  Prototype 
applications  for  IEEA  utilize  prior  research  on  Data-Driven  Documents  (D3),  a  representation- 
transparent  framework  for  rapid  development  of  online  data  visualizations  developed  by  the 
Stanford  Visualization  Group  (Bostock  et  al.,  2011).  D3  is  used  for  producing  dynamic, 
interactive  data  visualizations  in  web  browsers  and  is  the  successor  to  earlier  work  on  the 
Protovis  framework  (Keim  et  al.,  2010;  Bostock  &  Heer,  2009).  It  allows  for  direct  manipulation 
and  modification  of  any  elements  in  the  document  object  model  (DOM)  and  enables  smooth 
animation  and  user  interactions. 

Several  other  concepts  from  research  within  the  visual  analytics  community  are  also  relevant  to 
the  current  research  on  IEEA.  Data  reduction  techniques  like  those  described  by  Fodor  (2002, 
Holbrey  (2006)  and  Liu  et  al.  (2013)  are  necessary  because  computational  resources  will  always 
be  a  constraint  on  complex  decision  problems  like  those  encountered  in  systems  engineering 
analysis.  Problems  arise  in  computing,  storing,  and  transmitting  the  data  from  a  database  or 
local  storage  device  to  the  user  display.  Several  researchers  have  also  focused  on  the  problem 
of  how  to  visualize  large  sets  of  data  once  they  reach  the  user  display  (Inselberg,  2000; 
Andrews,  1972),  Ankerst  et  al.,  1998;  Delorme  &  Makeig,  2004).  Others  have  focused  more  on 
the  specific  types  of  visualizations  such  as  scatter  plots,  tree  maps  and  parallel  coordinate 
diagrams  (Chi,  2000;  Cleveland  &  McGill,  1984;  Carr  et  al,  1987;  Inselberg  &  Dimsdale,  1990; 
Keim,  2000). 

The  background  provided  by  this  prior  work  is  extremely  important  to  solving  the  problem  at 
hand,  but  Curry  (2017)  argues  that  for  decision  making  in  systems  engineering  applications  the 
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most  important  factor  is  arguably  whether  the  decision  maker  understands  and  can  find 
insights  in  the  multi-faceted  data  they  are  viewing.  Recent  work  on  multiple  coordinated 
visualizations  has  been  geared  towards  determining  how  user  displays  with  multiple  types  of 
information  can  be  used  to  extract  deeper  meaning  from  the  data  (Roberts,  2007;  Scherr,  2008; 
Andrienko  &  Andrienko,  2007).  This  is  a  potentially  useful  technique  for  the  types  of  systems 
engineering  interfaces  that  are  prototype  applications  generated  for  this  research. 

Many  engineers  make  decisions  about  constraints  on  performance  parameters  (outputs) 
without  considering  how  that  restricts  ranges  on  design  variables  (inputs).  As  an  example,  if  an 
engineer  sets  a  resolution  requirement  on  a  satellite  that  requires  a  certain  optical  aperture 
size,  this  will  likely  restrict  the  set  of  available  launch  vehicles  that  can  take  that  satellite  into 
orbit  due  to  fairing  size  restrictions.  Supplying  the  engineer  with  immediate  visual  feedback  on 
the  consequences  of  their  decisions  could  be  enabled  through  simultaneous  coordinated  views 
of  both  the  design  and  performance  spaces.  Curry  (2017)  points  out  that  enabling  users  to 
interact  with  their  data  through  visual  interfaces  of  this  type  is  an  area  of  active  research. 
Integrating  data  interaction,  algorithmic  analysis  and  advanced  engineering  methods  are 
important  aspects  for  current  research  effort  on  IEEA. 

Interactive  Visualization  for  Interactive  Epoch-Era  Analysis 

The  research  on  interactive  visualization  in  context  of  epoch-based  analysis  allowed  for 
exploration  and  demonstration  of  what  is  possible;  refinement  of  the  existing  visualizations  and 
development  of  new  ones  is  necessary  for  real-world  application.  Additional  expert  review  and 
more  extensive  user  testing  on  real-world  applications  would  likely  generate  new  ideas  for 
customized  visualization  types  and  interaction  methods.  Scalability  of  existing  visualizations 
requires  further  work,  such  as  more  scalable  versions  of  parallel  coordinate  diagrams  and  force 
directed  graphs.  Future  research  on  development  and  use  of  new  interface  types  such  as  touch 
interfaces  and  distributed  mobile  platforms  is  also  needed. 

Curry  et  al.  (2017)  and  Curry  (2017)  provide  details  on  the  visualization  prototypes  that  were 
explored  in  the  IEEA  research.  The  interactive  visualization  capability  is  intended  to  provide  a 
means  for  decision-makers  to  interactively  evaluate  the  performance  of  multiple  design 
alternatives  across  multiple  futures.  Such  capability  creates  opportunities  for  new  insights  at 
the  expense  of  a  potentially  large  and  complex  data  set  that  can  be  difficult  to  analyze.  The 
application  of  an  interactive  framework  allows  the  users  to  visualize  and  engage  with  the  data 
in  new  ways  that  can  facilitate  improved  comprehension  and  decision-making.  The  insights  that 
are  extracted  from  this  approach  allow  decision-makers  to  understand  the  characteristics  of 
designs  that  can  sustain  value  in  all  possible  futures,  through  passive  robustness  or  active 
changeability. 

An  example  visualization  for  IEEA  single  epoch  analysis  is  comparable  to  what  is  often  referred 
to  in  practice  as  tradespace  exploration  via  scatterplots.  Within  a  given  epoch,  a  scatterplot  of 
cost  (MAE)  versus  benefit  (MAU)  can  be  constructed  that  is  fixed  for  short-run  periods  of  stable 
context  and  needs  (i.e.,  an  epoch).  Typically,  decision-makers  want  to  identify  the  frontier  of 
Pareto  optimal  designs  or,  more  generally,  designs  that  are  "close  enough"  to  the  Pareto  front. 
Here  the  notion  of  "close  enough"  is  operationalized  through  Fuzzy  Pareto  Number  (FPN)  which 


Report  No.  SERC- 2017- TR- 104 


20 


Date  February  28,  2018 


is  used  to  quantify  distance  from  the  Pareto  Front  for  each  design  in  each  epoch.  FPN  is  a 
"within-epoch"  metric  and  its  score  for  a  given  design  will  change  from  epoch  to  epoch. 
Decision-makers  can  gain  insights  regarding  difficulty  of  a  particular  set  of  context  and  needs  by 
visualizing  how  points  move  in  the  design  space  as  the  epoch  and  FPN  values  change. 
Additional  insights  may  be  gained  from  interactively  filtering  the  design,  performance,  or  value 
variables.  This  can  be  performed  with  aid  of  a  filtering  application  to  allow  decision-makers  to 
interact  with  data  to  identify  designs  and  epochs  of  interest  (Figure  7).  It  allows  them  to  assign 
any  of  the  defined  variables  to  the  radius,  color  or  x-y  location  of  the  points  in  the  scatter  plot 
to  explore  the  data  in  four  dimensions  to  better  comprehend  the  behavior  of  the  designs. 


Figure  7.  Interactive  Filtering  Application  for  tradespace  exploration  for  the  offshore  ship  design  base  case 

Figure  7  illustrates  the  tradespace  for  the  offshore  ship  design  base  case  that  is  the  targeted 
contract  with  no  technical  requirements.  Hence,  at  this  initial  stage,  one  can  focus  on 
understanding  the  dynamics  of  the  underlying  system.  In  this  particular  case  study,  the  MAU 
degenerates  into  a  single  attribute  utility  function,  which  is  a  function  of  eco-friendliness,  even 
though  the  figure  indicates  a  multi-attribute  utility  function  on  a  general  basis.  The  interactive 
filtering  can  aid  in  visualizing  the  exploration  process  of  understanding  the  relative  significance 
of  individual  design  variables,  as  illustrated.  For  instance,  filtering  by  beam  and  length,  one  can 
see  that  relatively  slender  ships  tend  to  contribute  to  low  FPN  values.  However,  this  again 
makes  a  design  less  stable  in  the  water,  which  restricts  the  possibilities  of  retrofitting  heavy 
equipment  on  deck  without  intervening  with  the  main  hull.  Further,  one  can  directly  see  the 
trade-offs  of  adding  "design  for  changeability"  (DFC)  levels,  as  design  points  shift  right  in  the 
tradespace  with  increasing  DFC  due  to  increased  cost.  (DFC  level  is  a  proxy  for  degree  of 
modularity  and  ability  to  more  easily  change  the  ship;  operationalized  with  impacts  on 
structural  mass  and  interfaces.) 
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Demonstration  Cases 


The  IEEA  framework  was  tested  and  refined  using  two  demonstration  case  studies:  (1)  Space 
Tug,  a  multi-mission  orbital  transfer  vehicle;  and  (2)  Commercial  Offshore  Ship. 

Space  Tug  Orbital  Transfer  Vehicle 

Application  of  IEEA  to  a  case  study  for  a  multi-mission  on-orbit  servicing  vehicle  demonstrates 
key  concepts  and  prototype  interactive  visualizations.  A  space  tug  may  be  used  for  a  variety  of 
missions  including  observing,  servicing  or  retrieving  on-orbit  spacecraft.  The  original  case  study 
described  by  McManus  &  Schuman,  2003)  is,  at  first  glance,  a  seemingly  simple  trade  study,  but 
despite  the  simplicity  of  the  system  model  the  analysis  is  actually  nontrivial.  Fitzgerald  (2012) 
expanded  upon  this  case  as  a  demonstration  of  his  valuation  approach  for  strategic 
changeability  (VASC).  The  case  study  demonstrated  in  this  IMCSE  research,  replicates  the  one 
by  Fitzgerald,  which  provides  for  an  interesting  comparison  since  the  application  of  IEEA  leads 
to  different  insights  that  impact  previous  conclusions.  Various  visualization  prototypes  were 
developed  and  tested  in  this  case.  Application  of  IEEA  to  the  case  study  for  a  multi-mission  on- 
orbit  servicing  vehicle  demonstrates  key  concepts  and  prototype  interactive  visualizations  (see 
Curry  &  Ross,  2016  for  a  detailed  description  of  the  case). 

Commercial  Offshore  Ship 

This  case  applies  the  Interactive  Epoch-Era  Analysis  (IEEA)  framework  on  a  case  study  from 
commercial  offshore  ship  design,  incorporating  techniques  from  visual  analytics  such  as 
interactive  visualizations  to  gain  insight  from  large,  high-dimensional  data  sets  resulting  in 
improved  strategies  for  value  sustainment.  This  case  study  is  based  on  a  commercial  ship  case, 
developed  by  Rehn  et  al.  (2016);  a  more  detailed  discussion  of  the  case  setup  is  provided  in  that 
paper.  Offshore  ships,  in  contrast  to  traditional  deep-sea  cargo  ships,  are  designed  to  provide 
special  operational  services  typically  related  to  the  offshore  oil  and  gas  industry. 

A  discussion  of  the  case,  with  prototype  application  examples  and  discussion  of  the  capabilities 
and  insights,  can  be  found  in  Curry  et  al.  (2017).  The  new  prototype  visualizations  described 
are  motivated  by  a  need  to  address  design  questions  that  are  not  well-suited  for  analysis  with 
metrics  alone,  often  applied  in  other  EEA  case  studies,  such  as  Fuzzy  Pareto  Number  (FPN)  or 
Fuzzy  Normalized  Pareto  Trace  (fNPT).  For  the  offshore  ship  design  case,  this  includes  assessing 
the  trade-off  between  designs  optimized  to  target  the  primary  mission  versus  being  robust  for 
uncertain  subsequent  missions.  Further,  considerations  related  to  the  implementation  of 
interactive  visualization  applications,  such  as  scalability  and  latency,  are  discussed  emphasizing 
a  need  for  continued  research  on  methods  for  effectively  handling  large,  high-dimensional  data 
sets  in  design  of  complex  systems  under  uncertainty. 

Figure  8  illustrates  a  single  epoch  analysis  interactive,  linked  visualization.  The  goal  of  this 
visualization  is  equivalent  to  traditional  tradespace  exploration:  find  the  Pareto  efficient 
solutions,  gain  situational  awareness  of  tradeoffs,  on  an  epoch  by  epoch  basis  (here  96  epochs, 
with  41,204  designs  in  each).  Figure  9  illustrates  a  multi-epoch  analysis  interactive,  linked 
visualization.  The  goal  of  this  visualization  is  to  identify  designs  that  do  well  ACROSS  the  epoch 
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space  (i.e.  robust)  through  either  passive  (versatility)  or  active  (changeability)  means.  Over  4 
million  data  points  are  abstracted  in  this  visualization,  embodying  the  "show  the  high  level" 
part  of  the  visual  analytics  mantra.  Figure  10  illustrates  a  second  multi-epoch  analysis 
interactive,  linked  visualization.  This  one  embodies  the  "zoom,  filter"  part  of  the  visual  analytics 
mantra.  Whereas  the  prior  visualization  enabled  a  situational  awareness  of  the  data,  this  one 
allows  for  diving  deep  into  the  details,  in  particular  using  filers  to  determine  which  active  or 
passive  means  relate  to  the  achieved  robustness  across  the  epoch  space.  The  implementation 
leverages  both  OLAP  and  binned  aggregation  to  enable  real  time  (i.e.  low  latency)  interaction. 
The  4  million  data  points  have  been  reduced  to  ~40  thousand  via  the  aggregation.  Figure  11 
illustrates  a  single  era  analysis  interactive,  linked  visualization.  The  goal  is  to  evaluate  the  value 
of  design-strategy  pairs  for  sequences  of  future  epochs.  A  strategy  defines  how  change  options 
will  be  used  to  sustain  value  (e.g.  maximize  utility  or  efficiency).  The  coordinated  visualization 
allows  for  interactive  filtering  based  on  five  era-level  metrics  that  evaluate  temporal  aspects  of 
value  delivery  (i.e.  the  "shape"  of  the  value  over  time  curve,  the  aspects  of  which  may  matter 
differently  to  different  stakeholders;  e.g.  slope,  volatility,  etc.) 


Epoch  =  i 


(This  is  1  of  96  epochs) 


L  B  Or  Crino  POB  Power  LWI  MP  Fuel  Type  ROV  OP  PipeLay  OFC  Deck  Area  Owl  Speed  Acq  coat  MAU  MAE 

120m  30m  13m  OMT  400 ppl  15000 kW  300 MT  1  1  1  1  0  3  775.74 m2  1 6683  MT  18.756 kn  22857  rrtS  0.46309  0.43122 


Figure  8.  Single  Epoch  Analysis  (within  context/needs)  example  visualization  for  offshore  case 
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Figure  9.  Multi  Epoch  Analysis  (across  context/needs)  "show  the  high  level"  example  visualization  for 

commercial  offshore  case 


Filtered  subset 
of  designs  based 
on  epoch 
performance 


Figure  10.  Multi  Epoch  Analysis  (across  context/needs)  "zoom,  filter"  example  visualization  for  commercial 

offshore  case 
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Interactive  Time  Series  with  Filterirr 


Efficiency,  MAU,  Cash  Flows  Filters 


Filtered  subset  of  designs 
based  on  era  performance 


40K  design  points  x  20  yr  eras 
(6  mos  time  step)  =  1.6M  data  points  per  Era 


Figure  11.  Single  Era  Analysis  (across  time)  example  visualization  for  commercial  offshore  case 


Insofar  as  generalizability  of  the  results,  these  techniques  and  visualizations  (albeit  some 
degree  of  visualization  customization  is  ALWAYS  needed)  can  be  used  to  evaluate  the 
attractiveness  of  modularity  in  acquisition.  In  particular,  modularity  is  an  enabler  for  reducing 
the  switching  cost  of  a  system  between  alternative  design  or  operational  states.  This  reduced 
switch  cost  comes  at  the  expense  of  increased  design  costs  and  carrying  costs  (e.g.  the  interface 
infrastructure  may  entail  increased  physical  mass,  power,  and  structural  integrity  than  a  non- 
modular  interface).  Additionally,  the  cost  of  developing  and  maintaining  a  set  of  modules  to  be 
available  when  needed  must  be  addressed.  IEEA  can  be  used  to  not  only  provide  a  framework 
for  including  both  modular  and  non-modular  solutions  for  comparison  within  a  tradespace 
(which  helps  to  assess  the  lay-in,  opportunity,  and  carrying  costs  of  modularity),  but  also  a 
means  for  determining  the  costs  and  benefits  of  having  the  option  to  switch  between  states 
(e.g.  swapping  a  mission  payload  or  manufacturing  line).  Agility  is  related  to  the  speed  by  which 
a  system  can  respond  to  the  (potential)  impact  of  a  disturbance.  Modular  approaches  may 
reduce  the  switch  time  and  cost  of  response,  but  the  particulars  of  the  module  options,  as  well 
as  needed  changes  in  the  underlying  system  and  its  enterprise  should  be  evaluated  to  have  the 
proper  scope  of  tradeoffs  considered.  As  far  as  implementation  specifics,  both  modular  and 
non-modular  design  options  would  populate  the  tradespace,  and  module  swapping  "options" 
would  be  implemented  as  change  rules/mechanism  that  would  transform  the  tradespace  into  a 
tradespace  network.  The  multi-epoch  analysis  would  enable  determination  of  which  modules 
are  most  attractive  in  which  epochs.  Era  analysis  would  enable  evaluation  of  the  timing  aspect 
(i.e.  via  strategies  determining  when  "to  switch"  or  "not  to  switch"  and  how  the  costs  and 
benefits  of  those  would  relate  to  a  degree  of  agility  in  the  system). 
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Impact  Assessment:  Human-in-the-Loop  Software  Study 


Over  the  course  of  the  research  it  was  found  that  in  spite  of  wide  consensus  in  the  literature 
that  interactive  visualizations  enable  human  performance  gains,  there  were  no  primary  sources 
providing  evidence  that  supports  this  "generally  recognized  as  true"  claim.  Therefore,  in  order 
to  have  some  amount  of  evidence  supporting  this  basic  claim,  which  is  needed  for  this  research, 
a  controlled  human  subjects  experiment  was  designed  to  assess  whether  interactive 
visualization  improves  user  performance  for  design  problem  tasks.  A  primary  working 
hypotheses  of  this  research  is  that  the  creation  of  visual  analytic  applications  that  couple 
interactive  visualization  with  design  methods  (like  Epoch-Era  Analysis)  will  benefit  human 
performance.  Accordingly,  this  experiment  (introduced  in  the  prior  phase  research  report) 
required  subjects  to  answer  questions  related  to  a  simplified  engineering  design  problem 
equivalent  to  a  multi-epoch  analysis  problem.  Subjects  were  randomly  assigned  to  one  of  four 
treatment  groups  that  were  distinguished  by  the  type  of  data  representation  or  analysis  tool 
they  were  given  to  solve  the  problem.  It  was  anticipated  that  both  the  treatment  group  and 
individual  differences  between  subjects  impact  performance  as  measured  by  task  completion 
time  and  accuracy.  Individual  differences  as  discussed  in  this  study  refer  to  the  subjects' 
personality  traits  and  spatial  reasoning  ability.  Results  of  the  experiment  confirm  that  this  is 
indeed  the  case  and  that  performance  further  relates  to  the  task  type  in  question.  The  question 
under  consideration  in  this  experiment  is:  Does  interactive  visualization  improve  user 
performance  for  design  problem  tasks  and,  if  so,  what  are  the  relative  contributions  of 
representation,  interaction  or  other  factor  to  user  performance? 

There  are  two  components  to  this  question  that  must  be  tested.  First,  is  there  any  measurable 
impact  on  human  performance  when  individuals  are  engaged  in  performing  analysis  tasks 
commonly  associated  with  engineering  system  design  problems?  Second,  can  the  degree  to 
which  interaction,  visualization  or  other  factors,  such  as  individual  differences  in  the  users, 
affect  human  performance  be  isolated  and  identified.  For  the  purpose  of  this  study  human 
performance  is  measured  in  terms  of  both  the  speed  and  accuracy  with  which  a  subject  can 
complete  a  relevant  task.  A  primary  working  hypothesis  of  this  research  is  that  interactive 
visualization  will  improve  design  task  performance  for  either,  or  perhaps  both,  of  these  metrics, 
and  visual  representation  and  interaction  with  data  used  to  complete  the  task  will  impact 
performance  in  different  ways.  To  decouple  the  relative  contributions  of  representation  and 
interaction  a  2-by-2  factorial  experiment  with  a  total  of  4  treatment  groups  was  designed.  Each 
treatment  group  corresponds  to  a  different  analytical  tool  that  is  provided  to  test  subjects  to 
complete  a  design  task.  All  subjects  were  randomly  assigned  to  a  treatment  group  and  asked  to 
perform  analyze  a  surrogate  design  problem,  comprised  of  several  tasks,  that  is  a  simplified 
version  of  a  multi-epoch  analysis  problem.  Because  individual  differences  in  personality  or 
spatial  reasoning  ability  may  also  play  a  role  in  task  performance,  data  regarding  these  factors  is 
also  collected  from  participants  using  a  pre-test. 

A  controlled  human  subjects  experiment  was  deemed  an  appropriate  and  effective  approach 
for  testing  the  working  hypotheses.  By  controlling  for  whether  or  not  a  subject  is  given  a  data 
visualization  and/or  an  interactive  capability  designed  to  aid  them  in  solving  a  particular  task 
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the  factors  most  influential  to  performance  can  be  isolated.  While  the  individual  differences  of 
the  subject  volunteers  cannot  be  controlled  this  experiment  can  still  effectively  measure  the 
impact  of  these  factors  by  collecting  a  large  and  diverse  sample  set  of  participants. 

The  participants  in  this  study  were  drawn  from  volunteers  using  Amazon's  Mechanical  Turk 
(MTurk)  online  crowdsourcing  marketplace.  Crowdsourcing  the  evaluation  of  interactive  data 
visualizations  is  an  attractive  option  because  of  its  convenience,  ease  of  scalability  to  large 
numbers  of  participants,  and  relatively  low  cost.  Previous  research  studies  have  assessed  the 
viability  and  validity  of  using  MTurk  for  these  types  of  studies  and  provide  guidance  for  its 
application  (Heer  &  Bostock,  2010).  Several  other  studies  dealing  with  visualization  and  human 
performance  have  also  shown  that  MTurk  subjects  can  generate  positive  results  (Ziemkiewicz, 
2013;  Brown,  et  al.,  2014;  Ottley  et  al.,  2015;  Micallef  et  al.,  2012). 

Treatment  Groups 

The  two  primary  controlled  variables  in  this  experiment  were  the  presence  or  absence  of  (1)  an 
abstracted  representation  of  the  data  (e.g.  chart,  graph,  visualization)  and;  (2)  an  interactive 
capability  for  manipulating  or  engaging  with  the  data  in  some  way  (e.g.  filtering,  sorting).  It  was 
hypothesized  that  these  two  things  could  aid  a  subject's  understanding  of  a  problem 
differently.  To  decouple  relative  contributions  of  representation  and  interaction,  a  2-by-2 
factorial  experiment  with  a  total  of  4  treatment  groups  was  designed.  Each  treatment  group 
corresponds  to  a  different  analytical  tool  provided  to  test  subjects  to  complete  a  design  task. 
Figure  I2summarizes  the  four  analytic  tools  subjects  received  as  treatments  in  this  experiment. 


4  Experimental 
treatment  groups 

Figure  12  Experimental  Treatment  Groups 


Subjects  were  randomly  assigned  to  one  of  these  four  treatment  groups,  but  each  was  given 
identical  tasks  and  questions  to  answer. 


Treatment  A:  Non-interactive  Table 

This  group  can  be  considered  the  control  or  standard  treatment  group  in  this  study.  Subjects 
are  given  a  non-interactive  table  on  the  screen  Figure  13)  that  they  must  scroll  through  to 
determine  the  answer  to  task  questions. 
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Figure  13.  Treatment  A  -  Non-Interactive  Table 


Treatment  B:  Non-interactive  Table  +  Visualization 

This  group  extends  the  tool  used  for  treatment  A  to  include  a  non-interactive  static  graph.  The 
graph  is  a  stacked  and  grouped  bar  chart,  as  shown  in  Figure  14Error!  Reference  source  not 
found.. 
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Figure  14.  Treatment  B  -  Non  interactive  Table  +  Visualization 


Treatment  C:  Interactive  Table 

This  group  extends  the  tool  used  for  treatment  A  to  include  an  interactive  capability  as  shown 
in  Figure  14.  A  row  of  input  boxes  above  each  column  allows  regular  expressions  (e.g.  "<=20") 
to  be  entered  to  filter  the  data  set.  Clicking  on  the  column  headers  allows  the  data  to  be  sorted 
by  each  data  dimension. 
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Figure  15.  Treatment  C  -  Interactive  Table 


Treatment  D:  Interactive  Table  +  Visualization 

Treatment  group  D  combines  the  interactive  filtering  and  sorting  capability  of  group  C  and  an 
interactive  bar  chart  similar  to  the  static  bar  chart  in  treatment  group  B,  as  shown  in  Figure  16. 


Figure  16.  Treatment  D  -  Interactive  Table  +  Visualization 


Trial  Protocol 

Participants  were  asked  to  complete  a  3-part  evaluation.  The  first  part  of  this  study  asks 
participants  to  complete  a  standard  test  of  spatial  ability  called  the  "Paper  Folding  Test".  The 
second  part  of  the  study  asks  participants  to  complete  a  survey  that  evaluates  the  standard  Big 
5  Personality  traits  and  locus  of  control  commonly  used  in  social  science  research  studies.  The 
third  part  of  the  study  asks  participants  to  answer  several  questions  about  surrogate 
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engineering  design  tasks  using  one  of  4  possible  web-application  interfaces  depending  on  which 
experimental  treatment  group  they  have  been  assigned  into  randomly. 

The  possible  interfaces  are  either: 

(A)  a  plain  text  table  of  data; 

(B)  a  static  graph  or  visualization  of  the  data; 

(C)  a  Microsoft  Excel-like  interactive  table  of  data;  or 

(D)  an  interactive  visualization  of  the  data. 


Participants  may  exit  the  study  at  any  point.  Data  from  participants  that  exit  the  study  early 
will  not  be  included  in  the  final  analysis.  The  trial  protocol  is  shown  in  Figure  17. 
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Figure  17.  Trial  Protocol 


Experimental  Task 

In  the  experimental  task,  participants  were  asked  a  series  of  9  questions  about  the  design 
problem  they  are  given.  The  amount  of  data  required  to  answer  questions  varied  so  that  the 
impacts  of  overloading  a  subject's  working  memory  could  be  assessed.  Questions  were  defined 
by  3  different  task  types:  (1)  Filter  tasks  ask  users  to  identify  or  count  some  subset  of  the  data 
points;  (2)  Sorting  tasks  require  data  to  be  ordered  numerically  or  alphabetically  to  answer  the 
question;  and  (3)  Trend  task  require  the  identification  of  a  pattern  in  the  data  across  groups  or 
categories.  After  completion  of  the  9  questions  related  to  the  surrogate  design  task  the  final 
question  of  section  3  asked  subjects  to  rate  their  perceived  workload  while  performing  tasks. 

Findings 

This  experiment  was  conducted  to  better  understand  if  interactive  visualization  improves 
decision-making  for  design  problems  involving  complex  systems  tradeoffs.  The  experiment 
collected  data  from  human  subjects  working  with  various  web-based  design  tools  to  perform  a 
surrogate  design  tasks  for  a  simplified  car  design.  A  total  of  104  subjects  participated  in  the 
experiment,  with  26  participants  per  treatment  group. 

By  controlling  variables  associated  with  the  type  of  interface  provided,  the  experiment  allowed 
the  impacts  of  visualization  and  interaction  on  human  performance  to  be  decoupled  and 
evaluated.  Additional  detail  about  how  individual  differences,  such  as  subjects'  personality  and 
spatial  reasoning  ability,  influence  results  were  evaluated  using  data  collected  from  pre-test 
questionnaires.  A  summary  of  experiment  results  by  treatment  group  is  shown  in  Figure  18. 
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Accuracy 


Avg.  Completion  Time  Avg.  Cognitive  Load  Avg. 


Figure  18.  Summary  of  experiment  results  by  treatment  group  (source:  Curry,  2017) 


Curry  (2017)  enumerates  several  conclusions  that  can  be  drawn  from  the  experimental  results 
about  the  impact  on  task  accuracy,  completion  time  and  cognitive  load. 

Task  Accuracy 

•  The  type  of  interface  used  to  answer  design  task  questions  influence  task  accuracy. 
Specifically,  adding  a  graph  improves  accuracy  on  trend  observation  tasks  and  adding  an 
interactive  capability  improves  accuracy  on  sorting  tasks. 

•  Presence  of  a  graph  or  interactive  capability  becomes  more  important  as  the  number  of 
data  points  associated  with  the  task  increases. 

•  Accuracy  is  strongly  correlated  to  an  individual's  spatial  reasoning  ability  when  either  a 
graph  or  interactive  capability  is  present. 

Completion  time 

•  The  effects  of  adding  a  graph  are  small,  but  adding  an  interactive  capability  significantly 
reduces  the  time  required  to  solve  design  task  problems.  This  effect  is  most  prominent 
on  tasks  that  have  25  or  more  data  points  associated  with  them. 

•  There  is  no  clear  correlation  between  task  completion  time  and  individual  personality 
characteristics. 

Cognitive  Load 

•  Adding  a  graph  or  interactive  capability  reduces  cognitive  load,  but  adding  both  does 
not  further  reduce  the  perceived  load. 

•  Reported  drops  in  mental  and  temporal  demand  as  well  as  lower  levels  of  frustration 
are  the  primary  drivers  of  reduced  cognitive  load. 

•  Cognitive  load  has  a  strong  negative  correlation  with  spatial  reasoning  ability.  As  spatial 
reasoning  ability  increases  cognitive  load  decreases  significantly.  This  correlation  is 
even  stronger  when  an  interactive  capability  is  available. 

The  human-in-the-loop  experiment  demonstrates  the  relative  contributions  of  the  two  primary 
components  of  visual  analytic  systems  (representation  and  interaction)  and  how  they  are 
impacted  by  factors  such  as  task  type,  data  volume  and  individual  personality  differences. 
Because  the  surrogate  design  problem  used  in  the  experiment  closely  mirrors  the  challenges  of 
an  actual  multi-epoch  design  problem  the  results  can  offer  guidance  on  how  the  interactive 
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applications  for  IEEA  should  be  constructed  (e.g.,  knowing  how  the  objective  of  a  given  task 
maps  to  the  type  of  element  (e.g.  graph,  table);  the  nature  of  the  assistance  the  application  is 
expected  to  provide).  Further,  this  experiment  also  shows  that  improvements  in  accuracy  are 
strongly  correlated  to  subjects'  spatial  reasoning  abilities,  but  improvements  in  task  speed  are 
not.  Knowing  that  an  individual's  innate  characteristics  are  relevant  to  their  performance  opens 
up  possibilities  for  pre-screening  users  or,  alternatively,  providing  targeted  assistance  by 
allowing  interactive  applications  to  adapt  to  their  user.  The  results  of  this  experiment 
demonstrate  the  potential  benefits  that  IEEA  applications  can  provide  and  may  also  improve 
future  interactive  applications  intended  for  engineering  system  designers. 

Further  details  of  the  study,  analysis  of  data,  limitations,  and  findings  are  available  in  (Curry, 
2017).  Future  research  is  suggested  in  regard  to  additional  development  of  interactive 
visualizations,  adaptation  of  visualization  applications  to  individual  differences,  and 
backend/computing  related  techniques  and  tools. 


Summary 

Phase  5  built  on  prior  phase  work,  testing  and  refining  the  Interactive  Epoch-Era  Analysis  (IEEA) 
framework,  aimed  at  providing  a  means  for  analyzing  lifecycle  uncertainty  when  designing 
systems  for  sustained  value  delivery.  The  framework  was  tested  on  two  demonstration  cases: 
an  on-orbit  space  vehicle  case  and  a  commercial  offshore  ship  case.  The  two  cases  served  to 
demonstrate  key  concepts  and  prototype  interactive  visualizations.  The  latter  case  included 
deeper  analysis  of  options  at  the  epoch-level  for  changeability  as  well  as  era-level  analysis  of 
time-dependent  aspects  of  system  value  (Curry  &  Ross,  2017).  A  human-in-the-loop  impact 
assessment  experiment  that  decoupled  interaction  and  visualization  was  completed,  with  a 
total  of  104  subjects.  The  experiment  resulted  in  several  conclusions  concerning  the  impact  of 
interaction  and/or  visualization  on  task  accuracy,  completion  time  and  cognitive  load. 
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Model  Curation 


The  January  2015  IMCSE  Pathfinder  Workshop  participants  cited  model  curation  as  an 
important  topic  for  investigation  in  evolving  model-centric  engineering  (Rhodes  &  Ross,  2015). 
As  engineering  practice  becomes  increasingly  model-centric,  models  are  valuable  assets  for 
designing  and  evolving  systems.  New  leadership  roles  and  responsibilities  become  necessary  in 
model-centric  enterprises. 


Background 

Reymondet  et  al.  (2016)  investigates  considerations  for  curation  in  the  engineering  of  complex 
socio-technical  systems.  As  engineering  practice  becomes  increasingly  model-centric,  models 
are  valuable  assets  for  designing  and  evolving  systems,  and  continue  to  exist  throughout  the 
lifespan  of  a  program,  making  the  management  and  control  of  models  and  digital  artifacts 
imperative.  Maturing  an  approach  for  model  curation  in  the  systems  engineering  field  can 
leverage  work  of  other  curation  practices  in  other  fields. 

The  National  Academies  performed  a  study  and  published  a  report  on  digital  curation,  stressing 
the  importance  of  "active  management  and  enhancement"  and  discussing  the  importance  of 
preparing  the  workforce  (NAS,  2015).  According  to  that  report,  digital  curation  is  unique  in 
that  "the  immense  and  ever-increasing  quantities  of  material  to  be  curated,  the  need  for  active 
and  ongoing  management  in  a  contacted  of  continually  changing  uses  and  technology,  and  the 
great  diversity  of  organizational  contexts  in  which  curation  occurs".  Social  curation  is  also  a 
relevant  area  that  focuses  on  collaborative  sharing  of  Web  content  organized  around  one  or 
more  particular  themes  or  topics.  Duh  et  al.  (2012)  define  social  curation  as  "the  human 
process  of  remixing  social  media  content  for  the  purpose  of  further  consumption".  Patel  et  al. 
(2009)  examined  curation  of  CAD  engineering  models,  noting  some  specific  challenges.  The  first 
is  that  "the  information  to  be  dealt  with  are  diverse  and  particularly  complex,  including  product 
geometry,  finite  element  analysis  models,  manufacturing  process  models,  etc."  Another 
challenge  is  engineering  organizations  need  to  communicate  that  information  to  a  wide  range 
of  different  stakeholders,  each  with  different  information  needs  and  access  rights".  They  note 
that  the  purpose  of  the  information  may  be  varied.  Their  work  is  primarily  focuses  on  technical 
strategies  for  curation;  but  also  say  "there  is  a  need  for  best  practice  guidelines  and  cost- 
benefit  models  to  aid  in  choosing  appropriate  curation  strategies  since  the  business  of  deciding 
a  suitable  path  is  non-trivial  and  contingent  on  many  factors"  (Patel  et  al.,  2009). 

As  digital  transformation  progresses,  model-centric  enterprises  need  to  re-examine  the 
leadership  roles  and  responsibilities  that  will  enable  digital  model-based  engineering  practice.  A 
study  by  the  US  Interagency  Working  Group  describes  seven  challenges  that  organizations  may 
encounter  in  infusing  digital  model-based  engineering  (IAWG,  2017).  The  challenges  imply  new 
or  strengthened  leadership  capabilities,  many  that  relate  to  a  curation-type  role,  as  well  as  re¬ 
examining  basic  leadership  capabilities  under  the  digital  paradigm.  New  model  curation 
leadership  capabilities  have  potential  to  address  many  cited  challenges  related  to 
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organizational  and  cultural  hurdles,  contractual  and  data  management  practices,  security  for 
single  source  of  truth,  and  others. 

Need  for  curation  in  model-centric  engineering  stems  from  the  significant  increase  in  models 
and  model-related  assets  (libraries  of  data  sources,  techniques,  etc.).  Figure  19  illustrates  the 
many  decisions  one  would  need  to  make  in  using  models  on  a  complex  systems  endeavor,  such 
as  an  airport  collaborative  decision  making  system.  This  brings  into  question  who  owns  such 
assets  at  an  enterprise  level,  and  how  these  are  managed  and  used/reused  over  time. 


Architect  an  airport  collaborative  decision  making  system  (CDM) 

"share  real-time  flight  information  and  delegate  authority  to  sequence  departures, 
in  orderto  maximize  capacity  use  and  reduce  congestion" 
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•  What  models?  what  platforms?  analysis  techniques? 

•  What  model  trades? 

•  Where  are  sources  of  data?  Sources  of  models? 

•  What  about  composability  of  my  models? 


Figure  19  Many  Decisions  in  Using  Model-Centric  Environments  (Reymondet  et  al.,  2016) 


Effective  model  curation  necessitates  clarity  across  the  systems  community  in  characterizing 
and  handling  models.  It  requires  formalizing  knowledge  of  models  and  determining  a  distinctive 
set  of  model  characteristics  (purpose,  input/output  types,  logic,  assumption  types,  model 
incompatibilities,  etc.).  Various  types  of  models  have  been  enumerated  by  the  systems 
engineering  community,  but  there  appears  to  be  insufficient  attention  given  to  model  purpose 
itself.  This  is  discussed  in  more  detail  in  the  IMCSE  Phase  4  report  (Rhodes  &  Ross,  2017). 
Reymondet  et  al.  (2016)  introduces  the  envisioned  role  of  a  curator,  along  with  capabilities  may 
be  required  in  terms  of  understanding  models. 


Investigating  New  Leadership  Capabilities 

Under  the  premise  that  model-centric  environments  of  the  future  will  necessitate  specialized 
leadership  capabilities  and  competencies,  a  new  leadership  role  for  curation  has  been 
investigated.  As  the  model-centric  environments  become  increasingly  complex  and  critically 
important,  there  is  a  need  to  more  strategically  lead  and  manage  them.  Empirical  knowledge 
gathering  has  investigated  the  challenges  and  needs,  and  explored  the  potential  roles  and 
responsibilities  for  this  curation  role. 


RepoitNo.  SERC- 2017- TR- 104 


34 


Date  February  28,  2018 


A  model  curation  function  (performed  by  a  group  of  individuals)  will  have  the  objective  of 
sustaining  highest  possible  benefits  and  outcomes  from  the  collective  set  of  model  assets  and 
formal  curation  practices.  Extending  from  the  various  types  of  curation  roles  and  activities  of 
other  fields,  the  model  curator's  role  is  envisioned  to  include  a  number  of  major  responsibilities 
and  support  of  various  staff.  At  the  organization  level,  the  curator  may  organize  training  and 
special  projects  related  to  model-based  engineering.  The  model  curator  (curation  function) 
would  set  and  administer  model-related  policies  and  practices.  The  curator  would  ensure 
models  and  related  documents  are  authenticated,  preserved,  classified  and  organized 
accordingly  with  model  metadata  standards.  The  curator  may  own  the  data  management  for 
models  and  related  information,  or  oversee  the  ownership  by  other  individuals  or  organization. 
As  needed,  a  curator  would  meet  with  individuals  and  teams,  who  will  create,  use  and  re-use 
models,  helping  to  determine  a  useful  classification  of  both  individual  models  and  sets  of 
models. 


The  phrase  “active  management  and  enhancement"  was  chosen  to  distinguish  curation  from  simply 
collecting  and  storing  data  and  information. 

Active  management  denotes  planned,  systematic,  purposeful,  and  directed  actions  that  make  digital 
information  fit  for  a  purpose.  It  includes  coordinated  activities  that  allow  users  to  understand  and  exploit 
digital  information  assets  and  to  ensure  their  integrity  overtime. 

Active  management  also  refers  to  activities  that  ensure  that  digital  information  will  remain  discoverable, 
accessible,  and  useable  for  as  long  as  potential  users  have  a  need  or  a  right  to  use  it.  It  may  further 
involve  securing  digital  information  from  unauthorized  access. 

Information  management,  data  management,  data  stewardship,  data  governance,  and  digital  archiving  are 
related  terms  used  to  describe  processes  and  activities  that  overlap  with  curation ....  What  distinguishes 
curation  from  these  other  fields  is  its  emphasis  on  enhancing  the  value  of  information  assets  for 
current  and  future  use  and  its  attention  to  the  repurposing  and  reuse  of  information,  both  within 
and  beyond  the  context  in  which  it  was  first  created  or  collected. 


Preparing  the  Workforce  for  Digital  Curation,  National  Academies  Press,  2015 


Figure  20  National  Research  Council  Study  on  Digital  Curation 


Enterprise  Level  Model  Curation 

IMCSE  research  suggests  driving  factors  for  model  curation  and  a  curation  leadership  role  at  the 
enterprise  level.  Although  reuse  of  models  can  have  benefits,  the  reality  is  that  legacy  models 
are  not  widely  used  beyond  their  original  purpose.  Rouse  (2015)  stresses  that  the  wealth  of 
existing  models  is  often  not  used  because  of  a  lack  of  knowledge  of  these  resources  and  the 
difficulty  in  accessing  them.  The  lack  of  access  to  models,  mistrust  of  models,  and  perception  of 
legitimacy  of  models  are  all  barriers  in  model  reuse  and  longevity.  Since  model  expertise  is 
largely  resident  in  individuals,  the  ability  to  select  and  compose  sets  of  models  is  typically 
limited  to  the  original  use.  In  many  enterprises,  modeling  competency  is  distributed  across 
individuals  and  organizations,  rather  than  leveraged  at  the  enterprise  level.  Accordingly,  the 
lack  of  a  centralized  leadership  authority  results  in  models  being  owned  and  managed  only  at  a 
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local  level.  In  some  cases,  modeling  efforts  are  often  duplicated  across  programs.  Often, 
programs  that  lack  model  experts  accordingly  do  not  benefit  from  the  collected  wisdom  of  the 
enterprise.  Potential  benefits  to  other  programs  are  not  considered. 

The  digital  paradigm  shift  is  leading  toward  a  situation  where  models  are  becoming  increasingly 
valuable  and  ultimately  have  equal/greater  value  than  the  physical  assets.  Models  exist  at  all 
levels  of  an  enterprise  (individual,  program,  business  unit,  enterprise)  but  not  presently  well- 
managed  as  enterprise  collection.  There  is,  however,  the  need  to  distinguish  between  the 
models  that  can  be  elevated  to  the  enterprise  collection  (deemed  of  value  and  quality  to  be 
used  across  the  enterprise,  and  worthy  of  distribution  outside  the  enterprise  (by  exchange, 
loan,  sale...),  and  the  models  that  are  used  only  at  a  local  level  and/or  be  designed  to  be  used 
for  quick  studies,  etc.  Enterprise  level  models  will  need  to  have  a  "model  pedigree". 


Model  Pedigree 

Model  pedigree  was  first  described  as  "model  demographics"  by  Gass  &  Joel  (1980),  and  the 
term  pedigree  was  subsequently  used  by  Gass.  A  pedigree  contains  all  of  the  information  about 
a  model,  its  origins  and  use  over  time.  As  described  by  Gass  &  Joel  (1980),  the  purpose  is  to 
"enable  the  decision  maker  to  determine  the  model's  status  with  respect  to  past  achievements, 
theoretical  and  methodological  state  of  the  art,  and  the  expert  advice  that  went  into  its 
development".  While  model  documentation  is  typically  developed,  the  content  of  the  pedigree 
may  contain  information  not  always  included  in  engineering  model  documentation  materials. 
Given  that  IMCSE  research  on  model-centric  decision  making  has  shown  that  trust  is  a  key 
determinant  in  use  of  models,  a  pedigree  provides  information  that  engenders  trust. 


Envisioning  an  Executive  Leadership  Role 

Many  enterprises  have  established  new  leadership  positions,  such  as  a  Chief  Digital  Officer 
(CDO).  This  role  varies,  but  typically  is  a  change  agent  for  all  things  digital,  with  primary  focus 
on  creating  business  opportunities.  There  is  frequent  debate  over  the  overlap  with  other 
executive  roles  and  many  feel  this  role  may  go  away  over  time.  Chief  Data  Officer  (CDO)  and 
Chief  Analytics  Officer  (CAO)  are  other  newer  executive  roles,  given  the  rise  of  big  data  and 
data  analytics.  The  roles,  however,  are  not  sufficient  for  the  enterprises  that  depend  upon 
model-centric  engineering  practice  for  designing,  developing  and  operating  engineered 
systems,  as  the  emphasis  is  somewhat  different. 

As  enterprises  undergo  transformation  to  become  a  model-centric  enterprise,  the  model 
leadership  role  can  be  expected  to  evolve  over  time.  Figure  21  shows  a  hypothetical 
description  of  the  model  leadership  approach  and  enterprise  characteristics  relative  to  model¬ 
centric  transformation. 
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Figure  21  Leadership  Approach  Evolves  as  Enterprise  Transforms  to  Model-Centric  Enterprise  (hypothetical) 


Chief  Model  Curation  Officer  (postulated  new  role) 

The  continued  investigation  of  model  curation  in  this  phase  has  confirmed  the  need  for 
evolving  a  leadership  role  for  this  purpose,  which  this  report  will  refer  to  as  Chief  Model 
Curation  Officer  (CMCO)  for  purposes  of  discussion. 

The  CMCO  is  envisioned  as  an  enterprise-level  leadership  role  that  focuses  on  the  strategic 
management  of  the  enterprise  collection  of  model-based  assets  (Figure  22).  A  curator  of 
institutional  collections,  such  as  museum  curator,  offers  an  analogy  for  the  proposed 
responsibilities  of  the  model  curator.  The  museum  curator's  role  is  an  essential  one  where 
highly  knowledgeable  curators  oversee  collections  of  artwork  and  historic  items,  with  support 
from  archivists  who  appraise,  edit,  and  maintain  permanent  records  and  historically  valuable 
documents.  Specialists  and  technicians  are  also  involved  in  various  capacities.  The  museum 
curator  has  deep  knowledge  of  the  collection,  with  responsibility  for  putting  together 
purposeful  special  exhibits.  The  CMCO  is  envisioned  to  have  a  similar  executive-level  role 
overseeing  the  enterprise-level  model  collection,  as  well  as  having  strategic  responsibility  for 
composing  sets  of  models  for  special  purpose.  An  example  of  special  purpose  is  a  model-based 
demonstration  of  new  system  capability  in  support  of  a  competitive  bid  or  market  opportunity. 
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Research  in  progress  to  develop  and  validate  R&R,  including: 

•  Determine  model  accessions,  de-accessions,  acquisitions,  loans 

•  Responsible  for  policies/practices  and  model  pedigree  standard 

•  Set  guidance  for  use  (e.g.,  model  composition  strategies) 

•  Set  strategic  goals  for  model-centric  practice  and  environments 

•  Lead  role  for  composing  model  demonstrators 

The  museum  curator  has  deep  knowledge  of  the  collection,  with  responsibility  for  putting 
together  purposeful  special  exhibits.  The  CMCO  is  envisioned  to  have  a  similar  executive- 
level  role  overseeing  the  enterprise-level  model  collection,  as  well  as  having  strategic 
responsibility  for  composing  sets  of  models  for  special  purpose.  An  example  of  special 
purpose  is  a  model-based  demonstration  of  new  system  capability  in  support  of  a  competitive 
bid  or  market  opportunity. _ 


Figure  22  Exploring  the  Curation  Roles  &  Responsibilities  (R  &  R) 


The  role  of  the  CMCO  does  not  negate  the  need  for  model  ownership  and  management  at  the 
local  level.  Not  all  models  are  suited  for  a  collection-level  asset,  and  many  models  will  be  the 
result  of  maturation  of  a  local-level  asset  (e.g.,  developed  on  a  particular  program  or  in  a 
research  project).  The  CMCO  is  envisioned  as  the  gatekeeper  for  selection  of  models  elevated 
to  the  enterprise-level  model  collection  (model  accessioning),  as  described  in  Figure  23. 


Model  Accessioning 

•  Accessioning  is  the  process  of  officially  accepting  models  (and  sets  of  models)  into  the 
enterprise  level  model  collection. 

-  An  accession  is  the  acquisition  of  a  single  model  or  set  of  models,  from  one  source,  under  one 
type  of  transaction  (elevated  to  global,  exchange,  purpose...),  as  of  a  single  date 

•  Accessioning  establishes  legal  ownership,  IP,  validity  of  pedigree,  etc. 

-  Developed  or  acquired?  Or  existing  local  asset  elevated  to  global  asset?  Adapted  from  open 
source?  Purchase  or  exchange? 

•  Condition  at  accession  recorded,  establishing  initial  pedigree 

-  Model  and  pedigree  info  recorded  with  unique  identifier 

-  Placed  under  CM 


Figure  23  Example  Role:  Model  Accessioning 


Potential  Forms 

During  this  phase  of  research,  continued  knowledge  gathering  and  semi-structured  interviews 
with  executives  were  conducted  to  formulate  a  preliminary  set  of  potential  forms  for  a  model 
curation  function.  Figure  24  describes  seven  alternate  forms  for  an  enterprise  to  execute  the 
role  and  responsibilities  of  an  enterprise  model  curator  function.  These  forms  are:  centralized  - 
top  tier,  centralized  -  dotted  line,  franchised,  collaborative,  dual  hat,  delegated,  and 
outsourced.  Further  research  is  needed  to  investigate,  validate  and  refine  this  initial  taxonomy. 
Over  time,  studies  are  needed  to  understand  how  effective  these  forms  are  for  various 
enterprises,  and  under  which  conditions  one  might  chose  the  form. 
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Form 

Implementation 

Under  what  conditions 

Centralized  - 
Top  Tier 

CMCO  is  a  top  tier 
executive  reporting 
directly  to  CEO 

High  performing  model-centric  engineering  enterprise  or  has 
a  very  aggressive  goal  to  become  one.  Culture  has  fully 
embraced  model-centric  engineering  across  the  entire 
enterprise,  and  believes  model  leadership  is  key  to 
competitive  advantage  and  to  innovation 

Centralized  - 

Dotted  Line 

CMCO  has  enterprise 
level  authority,  “dotted 
line"  reporting  to  CEO 

The  enterprise  is  rapidly  becoming  a  high  performing  model¬ 
centric  engineering  enterprise.  The  CMCO  needs  enterprise 
level  authority  to  make  and  implement  strategic  decisions, 
but  is  not  ready  to  make  the  CMCO  a  full  member  of  the 
executive  management  team. 

Franchised 

Enterprise  segments 
have  CMCOs, 
conforming  to 
enterprise  defined  policy 
and  role. 

For  very  large  enterprises  with  varied  cultures  within 
enterprise  segments,  it  can  be  difficult  to  have  one  top-tier 
CMCO.  A  franchised  form  enables  the  enterprise  to  establish 
the  role  and  responsibilities  at  enterprise  level.  CMCOs  are 
appointed  in  each  enterprise  segment  reporting  to  its  top 
executive,  conforming  to  enterprise  defined  policy  and  role. 

Collaborative 

Virtual  CMCO  role 
through  collaborative 
committee 

The  enterprise  does  not  wish  to  appoint  a  single  individual  to 
the  CMCO  role  at  this  time,  but  recognizes  the  need  to  have 
model  curation  capabilities.  An  appointed  collaborative 
committee  provides  a  "virtual  CMCO"  by  serving  as  a 
strategy-setting  and  implementation  oversight  body.  An 
instance  where  this  might  be  a  preferred  form  is  an 
enterprise  that  is  comprised  of  many  newer  acquisitions 
that  have  strong  heritage  culture  and  processes. 

Dual  Hat 

CMCO  is  one  of  two 
roles  played  by  an 
executive 

An  enterprise  that  is  working  toward  becoming  a  model¬ 
centric  enterprise  but  still  uses  legacy  approaches  is  likely 
not  ready  to  make  the  investment  needed  to  appoint  an 
enterprise  CMCO.  A  dual  hat  form,  while  challenging  to 
perform,  enables  the  enterprise  to  allocate  specific  model 
curation  responsibilities  to  an  existing  leader  (e.g.,  CIO,  CTO, 
CDO). 

Delegated 

CMCO  responsibilities 
are  delegated  to  a 
committee,  to  one  or 
more  individuals 

Similar  to  "dual  hat"  but  the  role  is  not  specifically 
delegated;  the  responsibilities  are  delegated  as  add-ons  to 
an  existing  role,  or  to  the  responsibilities  of  a  standing 
committee.  This  implementation  form  is  weaker  than  the 
dual  hat,  but  does  acknowledge  these  responsibilities  as 
specific  to  MC,  as  a  precursor  to  a  formal  role. 

Outsourced 

CMCO  role  is  performed 
by  an  external  hire 

Temporary  hiring  of  an  outside  CMCO  may  be  the  only 
option  available  to  an  enterprise  that  recognizes  the  need 
for  the  MC  role  but  does  not  have  expertise  or  capacity  to 
staff  it  from  within.  It  might  also  be  a  form  used  by  an 
enterprise  that  is  not  convinced  of  the  value  of  having  an 
CMCO,  but  wants  to  have  trial  implementation. 

Figure  24  Organizational  Forms  for  Chief  Model  Curation  Officer 

Report  No.  SERC- 2017- TR- 104 


39 


Date  February  28,  2018 


Model  Curation  Lexicon 

Table  1  enumerates  a  lexicon,  which  is  evolving  through  discussions  with  research  stakeholders 
and  investigation  of  other  fields. 

Table  1  Model  Curation  Lexicon  (continues  to  evolve) 

Chief  Model  Curation 

Officer 

A  designated  professional,  authoritative  role  -  at  the  executive  level  of  the 
enterprise  -  entrusted  with  the  ownership,  tracking  and  use  of  model-based 
assets. 

Model  Accessioning 

The  formal  process  of  accepting  and  recording  a  model  as  a  collection  object 
in  the  enterprise  level  model  portfolio.  Accessioning  addresses  the  legal,  IP 
and  ethical  issues  in  model  acquisition  and  development. 

Model  Acquisition 

The  act  of  acquiring  a  model  through  an  arrangement  with  the  model  owner 
(e.g.,  through  purchase,  trade,  or  other  business  transaction). 

Model  Cataloging 

The  formal  process  of  making  a  model  available  for  use  through  recording  it  in 
a  catalog  or  directory,  and  tracking  it  throughout  the  model  lifecycle. 

Model  Collection 

The  collection  of  model-based  assets  that  is  possessed  by  an  enterprise, 
including  those  developed  by  the  enterprise,  acquired  by  the  enterprise,  and 
temporarily  resident  in  the  collection  (e.g.,  leased,  on  loan). 

Model  Collection 

Object 

A  model  or  model-related  object  that  is  a  unique  asset  in  the  enterprise's 
collection.  An  object  is  assigned  a  unique  identifier. 

Model  Curator 

A  designated  professional  role  entrusted  with  the  ownership,  tracking  and  use 
of  model  collection  objects,  and  possessing  designated  authorities  for 
managing  and  controlling  models. 

Model  Composition 

The  process  of  composing  models  and  model-related  information  that 
provides  collective  value  beyond  the  individual  models. 

Model  Composability 

The  characteristic  of  an  interrelated  set  of  models  that  enables  them  to  be 
combined  in  accordance  with  given  modeling  formalisms. 

Model  De-accessioning 

The  formal  process  of  removing  a  model  as  a  collection  object  from  the 
enterprise  level  model  portfolio. 

Model  Demonstrator 

A  composed  set  of  models  with  interactive  interfaces  for  the  purpose  of 
demonstrating  context-specific  capability.  Demonstrators  enable  the  modeled 
system  to  be  experienced  by  an  individual  through  conveying  cogent 
information  and  where  possible,  the  use  of  interactive  interfaces. 

Model  Loan 

The  act  of  temporarily  acquiring  a  model  through  an  agreement  whereby  the 
model  owner  agrees  to  share  the  model  with  the  model  acquirer  for  a 
specified  time  and  specified  terms  (e.g.,  terms  of  use,  remuneration,  etc.). 

Model  Metadata 

Descriptive  metadata  is  contextual  data  about  the  model  object(s).  Metadata 
documents  characteristics  and  used  for  indexing,  discovering,  identification. 
Provides  user  discovery  of,  access  to,  and  management  of  an  object. 

Model  Pedigree 

Model-associated  information  that  describes  model  origin,  development 
process,  originators  and  developers,  assumptions,  expert  knowledge,  model 
enhancements,  investment  costs,  versions,  change  history,  etc. 
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Discussion  and  Future  Research 


The  research  has  identified  sociotechnical  leadership  capabilities  that  provide  the  ability  to 
execute  digital  model-based  engineering  at  the  program  and  enterprise  levels.  The  next  phase 
of  research  seeks  to  mature  a  framework  for  assessing  model-centric  enterprise  capabilities. 
This  framework  will  be  developed  using  an  evidence-based  approach,  which  integrates 
empirical  research  evidence,  decision  maker/practitioner's  expertise,  and  situational 
perspectives  (values,  needs,  stakeholder  preferences  such  as  risk  tolerance).  The  assessment 
framework  will  accommodate  assessing  capability  respective  to  the  state  of  transformation 
from  traditional  to  fully  digital  model-based  engineering.  Model-centric  enterprise  capabilities 
identified  in  prior  phase  research  include  model  composability,  transparency,  accession 
practices,  model  valuation  practices,  model  trust  and  others.  Ultimately,  having  a  common 
framework  for  model-centric  enterprise  capabilities  enables  better  understanding  of  effective 
practices,  accessing  transformation  progress  and  generation  of  evidence  through  systematic 
study,  providing  benefits  across  the  systems  community. 


Summary 

Under  the  premise  that  model-centric  environments  of  the  future  will  necessitate  specialized 
leadership  and  competencies,  a  new  leadership  role  for  curation  was  further  investigated.  In 
this  phase,  alternative  stages  were  identified  for  organizations  transforming  under  the  digital 
paradigm.  Specific  needs  were  identified  for  establishing  leadership  and  practices  for  model 
curation.  Investigation  resulted  in  seven  different  types  of  organizations  for  implementing 
model  curation,  based  on  different  needs  and  organization  forms.  Research  findings  were 
shared  with  research  stakeholders  as  a  preliminary  validation  of  needs  and  emerging  practices. 
The  need  for  an  instrument  for  organizations  to  assess  their  model  leadership  capabilities  and 
risks  was  confirmed  in  a  technical  exchange  workshop  and  other  meetings  with  research 
stakeholders. 
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Conclusion 


IMCSE  research  seeks  to  inform  and  contribute  new  knowledge,  processes,  methods  and  tools 
to  improve  the  interactivity  of  humans  and  models  in  support  of  systems  decision  making.  The 
research  is  grounded  in  two  assumptions.  The  first  is  that  systems  success  depends  upon 
effective  "human-model  teaming"  and  the  second  is  that  specialized  leadership  and 
competencies  will  be  required  to  realize  the  vision  of  model-centric  engineering. 

Phase  5  research  included  three  areas  of  focus,  which  built  upon  the  prior  phase  research. 
These  are: 

Human-Model  Interaction.  The  empirical  study  on  model-centric  decision  making  was 
completed  with  30  experts,  resulting  in  findings  and  preliminary  guiding  heuristics.  Outcomes 
of  the  investigation  of  human-model  interaction  were  integrated  and  shared  in  state  of  the 
practice  papers  and  presentations.  Heuristics  were  further  refined  and  validated,  with  four 
proposed  application  areas.  A  technical  exchange  workshop  was  held  to  gather  practitioner 
feedback  on  study  results  and  directions  for  transitioning  heuristics  to  practice. 

Interactive  Epoch-Era  Analysis  (IEEA).  The  IEEA  framework  and  prototypes  were  applied  in  a 
commercial  ship  design  case  to  test  applicability  to  a  non-defense  application  and  analysis  of 
changeability  options.  The  designed  experiment  on  impacts  of  decoupled  visualization  and 
interaction  was  completed.  The  framework,  prototypes,  case-based  impact  studies,  and 
experiment  results  were  completed. 

Curation  of  Model-Centric  Environments  (Enterprises).  Model  curation  was  further  explored 
through  literature  and  interaction  with  research  stakeholders.  Investigation  areas  included 
alternative  forms  for  curation  leadership,  curation  practices,  and  data/model  pedigree. 
Preliminary  exploration  of  a  concept  for  an  assessment  instrument  for  curation  capabilities  was 
conducted.  A  technical  exchange  session  was  held  to  gather  feedback  on  interim  research, 
identify  potential  transition  opportunities,  and  elicit  information  on  state  of  practice  and  needs 
for  the  future. 

IMCSE  research  has  been  presented  and  discussed  with  practitioners  and  sponsors  in  numerous 
research  meetings  and  workshops,  as  well  as  with  other  researchers  in  the  systems  community. 
These  activities  have  raised  the  awareness  of  challenges  and  needs  surrounding  human-model 
interactivity,  and  there  is  a  growing  community  of  interest  within  the  SERC  and  the  larger 
systems  community. 

IMCSE  research  will  continue  in  Phase  6,  with  two  areas  of  focus.  The  first  is  guiding  principles 
and  patterns  for  human-model  interaction.  The  second  area  is  a  framework  for  assessing 
model-centric  enterprise  capabilities. 
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